Draft: Test if empty branches in the output file exist

Add a macro which loops over all defined branches according to the BranchList folder and checks is empty branches exist. Empty branches indicate a problem. Add test to the CI which execute the macro for all created output files.

changed milestone to %OCT23

added Build System CI/CD labels

requested review from @p.-a.loizeau

assigned to @p.-a.loizeau

marked this merge request as draft

This MR should only be merged after the creation of the MvdDigi is fixed. Currently the the CI pipelines should fail since new test were added which checks if empty branches in the output files exist. In case empty branches are found the test fails since this is an obvious bug. All existing branches should contain data.

approved this merge request

added 20 commits

• b80b35a3...5c7893f5 - 16 commits from branch computing:master
• 84ce846a - Fix generation of MvdDigi in event based mode
• c47fff72 - Add back event mode in MVD cluster finder
• 8eb8f942 - Fix the reconstruction topologies at several places
• 96b4166e - Test if empty branches in the output file exist

Compare with previous version

added 113 commits

• 96b4166e...15116b14 - 112 commits from branch computing:master
• e508ae33 - Test if empty branches in the output file exist

Compare with previous version

added 1 commit

• 03b5d7ca - Test if empty branches in the output file exist

Compare with previous version

added 54 commits

• 03b5d7ca...ea9125b8 - 53 commits from branch computing:master
• dbac3f72 - Test if empty branches in the output file exist

Compare with previous version

added 34 commits

• dbac3f72...78e217d6 - 33 commits from branch computing:master
• 02fa7efe - Test if empty branches in the output file exist

Compare with previous version

added 5 commits

• 02fa7efe...b5b9540c - 4 commits from branch computing:master
• 11a0246d - Test if empty branches in the output file exist

Compare with previous version

removed milestone %OCT23

added 179 commits

• 11a0246d...39c00e70 - 178 commits from branch computing:master
• a2db261a - Test if empty branches in the output file exist

Compare with previous version

I did not think about it before but the two remaining failed tests kind of make sense if the "no empty branch" convention was not made explicit before: in both case some Muon setup reco test fails with

[ERROR] Branch TrdTrack is empty

What is still strange is that this seems to happen only on CentOs while I would expect it to be general to all jobs

In my opinion this is a statistical problem. If there is not enough data there is probably no track. Can you check for the other systems if there is the TrdTrack branch at all?

For the other systems the log is truncated before this line as it succeeds

Will try to run it in a virgo container or on mFLES to see

Indeed when testing with the weekly mode it worked also for CentOs (if I did not mix up the OSes, was a bit complicated as tehre is not shortcut anymore for the centos7 on virgo)

Switching to DAILY mode (by commenting out the cmake WEEKLY option), I can reproduce the error.
When comparing the logs, I get some completely different logs already at transport level, so I am wondering now if the problem is not simply that the random generator behaves differently under centos from vae23, leading to non-comparable results.

If you want to have a look:

meld /lustre/cbm/users/ploizeau/apptainer/empties_*_nov22p1_daylies.txt