Online raw digi loader

• Allows a similar conversion/inspection at Raw Digis level as was done before at DigiEvents level.
• Tool macro was used to cross check the changes in mapping for mTOF did not have side-effects for the 2022 data
  • True between "old unpacker" output trees before/after the mapping changes
  • True between "online unpacker" output files before/after the mapping changes
  • False between "old unpacker" and "online unpacker" in both case, beyond just missing leaves like Pulse ones
    => I think we will need a further cross-check with both the mCBM 2022 and 2024 data, as my feeling from these test is that the online unpacker does not reproduces 1:1 the old unpacker
    => May be due to some parameters getting out of sync or some missing/extra corrections (walks?)
    => But the unpacking stage of online and offline should never differ in any of the Time/TOT-Charge/Address fields of the output digis!
• Also include a fix to the Digi Events inspection macro to allow stopping after a given number of TS

Usage example:

./build_local/bin/cbmreco --steps Unpack -i /scratch/mcbm_data/mcbm2022/2391_first20Ts.tsa --param-dir ./parameters/online/ -O RawDigi -o crosscheck_online_2391.out
cd macro/run
root -l -b -q 'run_convert_online_rawdigis.C("../../crosscheck_online_2391.out", "data/online_2391.root")'
cd -
cd build_local
ctest -R mcbm_unpack_2391
cd -
cd macro/tools
root -l -b -q 'TreeCompareAuto.C("../../build_local/macro/run/data/2391_first20Ts.digi.root", "../run/data/online_2391.root")'

Probably of interest to @v.friese, @a.toia, @n.herrmann, @fweig, @d.smith, @dschledt

Unpackers validation:

• BMON
• STS
• TRD2D
• TRD1D
• TOF
• RICH

added Online QA Reconstruction mCBM labels

requested review from @v.friese

assigned to @v.friese

Dear @f.uhlig, @p.-a.loizeau, @v.friese,

you have been identified as code owner of at least one file which was changed with this merge request.

Please check the changes and approve them or request changes.

added CodeOwners label

For now I find the following differences after unpacking and converting single detectors in the first 20TS of run 2391:

• BMON: charge identical, address almost identical, time different
• STS: address, time and charge different, Pulser not there
• TRD 1D + 2D: time is identical, Info and charge different (failing Kolmogorov)
• TOF: address and charge identical, time different
• RICH: time and charge identical, address different

Could still be the conversion or the comparison which are not valid, but then I would expect all branches to fail (or the same set of branches for each detector) I will try to have a more in-depth look to one of the detectors tomorrow or next Tuesday

@a.toia Here are the commands I used to do the comparison with the fork taken as source of this MR (same info inside the attached file in case it is easier to download it: legacy_online_unpackers_comp.txt)

./build_local/bin/cbmreco --steps Unpack -i /scratch/mcbm_data/mcbm2022/2391_first20Ts.tsa --param-dir ./parameters/online/ -O RawDigi -o crosscheck_online_2391_STS.out -s STS
./build_local/bin/cbmreco --steps Unpack -i /scratch/mcbm_data/mcbm2022/2391_first20Ts.tsa --param-dir ./parameters/online/ -O RawDigi -o crosscheck_online_2391_BMON.out -s BMON
./build_local/bin/cbmreco --steps Unpack -i /scratch/mcbm_data/mcbm2022/2391_first20Ts.tsa --param-dir ./parameters/online/ -O RawDigi -o crosscheck_online_2391_TRD1D2D.out -s TRD TRD2D
./build_local/bin/cbmreco --steps Unpack -i /scratch/mcbm_data/mcbm2022/2391_first20Ts.tsa --param-dir ./parameters/online/ -O RawDigi -o crosscheck_online_2391_TOF.out -s TOF
./build_local/bin/cbmreco --steps Unpack -i /scratch/mcbm_data/mcbm2022/2391_first20Ts.tsa --param-dir ./parameters/online/ -O RawDigi -o crosscheck_online_2391_RICH.out -s RICH

root -l -b -q 'macro/run/run_convert_online_rawdigis.C("crosscheck_online_2391_STS.out", "online_2391_STS.root")'
root -l -b -q 'macro/run/run_convert_online_rawdigis.C("crosscheck_online_2391_BMON.out", "online_2391_BMON.root")'
root -l -b -q 'macro/run/run_convert_online_rawdigis.C("crosscheck_online_2391_TRD1D2D.out", "online_2391_TRD1D2D.root")'
root -l -b -q 'macro/run/run_convert_online_rawdigis.C("crosscheck_online_2391_TOF.out", "online_2391_TOF.root")'
root -l -b -q 'macro/run/run_convert_online_rawdigis.C("crosscheck_online_2391_RICH.out", "online_2391_RICH.root")'

# [ Comment all detectors at lines 493-500 of the macro ]
root -l -b -q 'macro/run/run_unpack_tsa.C("/scratch/mcbm_data/mcbm2022/2391_first20Ts.tsa",2391,"",-1,false,"./")'
mv 2391_first20Ts.digi.root 2391_first20Ts_STS.digi.root
rm -rf qa 2391_first20Ts.perf.root 2391_first20Ts.mon.rich.root
# [ Comment all detectors at lines 493-500 of the macro ]
root -l -b -q 'macro/run/run_unpack_tsa.C("/scratch/mcbm_data/mcbm2022/2391_first20Ts.tsa",2391,"",-1,false,"./")'
mv 2391_first20Ts.digi.root 2391_first20Ts_BMON.digi.root
rm -rf qa 2391_first20Ts.perf.root 2391_first20Ts.mon.rich.root
# [ Comment all detectors at lines 493-500 of the macro ]
root -l -b -q 'macro/run/run_unpack_tsa.C("/scratch/mcbm_data/mcbm2022/2391_first20Ts.tsa",2391,"",-1,false,"./")'
mv 2391_first20Ts.digi.root 2391_first20Ts_TRD1D2D.digi.root
rm -rf qa 2391_first20Ts.perf.root 2391_first20Ts.mon.rich.root
# [ Comment all detectors at lines 493-500 of the macro ]
root -l -b -q 'macro/run/run_unpack_tsa.C("/scratch/mcbm_data/mcbm2022/2391_first20Ts.tsa",2391,"",-1,false,"./")'
mv 2391_first20Ts.digi.root 2391_first20Ts_TOF.digi.root
rm -rf qa 2391_first20Ts.perf.root 2391_first20Ts.mon.rich.root
# [ Comment all detectors at lines 493-500 of the macro ]
root -l -b -q 'macro/run/run_unpack_tsa.C("/scratch/mcbm_data/mcbm2022/2391_first20Ts.tsa",2391,"",-1,false,"./")'
mv 2391_first20Ts.digi.root 2391_first20Ts_RICH.digi.root
rm -rf qa 2391_first20Ts.perf.root 2391_first20Ts.mon.rich.root

root -l -b -q 'macro/tools/TreeCompareAuto.C("2391_first20Ts_BMON.digi.root", "online_2391_BMON.root")'
root -l -b -q 'macro/tools/TreeCompareAuto.C("2391_first20Ts_STS.digi.root", "online_2391_STS.root")'
root -l -b -q 'macro/tools/TreeCompareAuto.C("2391_first20Ts_TRD1D2D.digi.root", "online_2391_TRD1D2D.root")'
root -l -b -q 'macro/tools/TreeCompareAuto.C("2391_first20Ts_TOF.digi.root", "online_2391_TOF.root")'
root -l -b -q 'macro/tools/TreeCompareAuto.C("2391_first20Ts_RICH.digi.root", "online_2391_RICH.root")'

@v.friese As it seems to work for STS, I think the conversion stage is more or less working as expected, so I would tend to close this thread and not wait for the feedback of all detectors in this MR

@dschledt reports in !1763 (merged) that after fixing a bug in the online version of the TRD1D unpacker the data are matching modulosome rounding error on the time field

I've now also checked 2391_first20Ts.tsa and it is completely equal for 5 timeslices between online and offline. No rounding differences here, but there is also no time reconstruction so only integer times.

The RICH unpackers are also identical (without !1738 (merged)),
the difference was that the offline unpacker dropped overlapping MS by default, while online processed those.
Adding richconfig->SetDoIgnoreOverlappMs(false); to the offline unpacker macro fixes this.

Tested on files 2391_first20Ts.tsa and 2511_en13_1_0002.tsa. In both cases:

Tested leaves:                    3
Leaves with identical entries:    1
Leaves with identical histograms: 2
Leaves with kolmo histograms:     0

the difference was that the offline unpacker dropped overlapping MS by default, while online processed those.
Adding richconfig->SetDoIgnoreOverlappMs(false); to the offline unpacker macro fixes this.

nice observation, did not thought/remember about this
That may actually explain most of the differences for all detectors

I will try to have a second look with the proper settings on Monday

Enabling the overlap MS indeed makes the RICH and TOF match when unpacking only the first TS. When trying to unpack the first 20 TS I still get some differences reported.

After some modifications in the comparison macro, it seems the main problem is that even when switching off the "IgnoreOverlap" flag, there are more entries in the online Tree than in the offline Tree.
I will try to investigate a bit more and push the expanded comparison macro after lunch

I had the same problem when I did my comparison. I then changed this line to:

- auto nrMsToLoop = fDoIgnoreOverlappMs ? fNrCoreMsPerTs : fNrMsPerTs;
+ auto nrMsToLoop = fNrMsPerTs;

After that the number of entries were equal

Correct, we do not currently support timeslice overlaps in the online software.

From what I understand the online software always unpack the overlap microslice. So if a proper handling of the overlap is really missing, I would suspect that the events in the overlap are duplicated (once in the overlap MS of current TS and once in the first MS of the next)

@dschledt As far as I can see with a bit of debug printout, this is probably only the case for TRD1D: the common template method CbmRecoUnpackConfig::SetAlgo is overloaded in CbmTrdUnpackConfig but the overload is missing the line propagating the "IgnoreOverlap" flag to the algo

I will add a fix for this in this MR

After fixing the flag propagation in the offline unpacker of TRD, a similar problem in the offline unpacker of BMON and made the unpacking macro more flexible to switch the overlap one and off, I now get for the "first TS" case:

• identical numbers for TOF, RICH and TRD1D
• similar numbers for STS and MUCH, which from @a.toia comments are probably due to the duplicates rejection (as there are more digis online than offline
• a small difference of 3 digis for BMON which I do not understand (471842 offline vs 471855 online)
• a difference of around 80 000 missing digis online for TRD2D (1 163 865 vs 1 081 078, so around 6-8 %). @a.bercuci or @d.smith, could this be due to the different call to FinalizeComponent() in the online version? Or do you remember anything else from !1156 (merged) or !1388 (merged) which could explain it?

Similar observations (numbers scaled by 20 but ratios more or less equivalent) hold for the case of unpacking the first 20 TS

The FinalizeComponent() issue definitely leads to a difference between offline and online results for 2D. This is unavoidable. Whether this explains the difference entirely I cannot say, but it is definitely one component.