Skip to content

QA-Checker Framework

Sergei Zharko requested to merge s.zharko/cbmroot:qa-checker into master

This merge request introduces the QA-Checker Framework, which provides a scheme for comparing ROOT-files, obtained with different versions of the CbmRoot. In general case, several versions of QA-results can be compared at the same time. The results can be kept in different ROOT files for different setups (datasets). The comparison method can be specified to point-to-point test, ratio-test, statistical test depending on the type of the ROOT object and defined methods. New comparison methods can be introduced and run simultaneously.

Framework introduction in core/qa/checker (namespace cbm::qa::checker)
  • Core: core checker class, defines the framework logic and user interface
  • ObjectDB: database of object names, processed in the framework (file-object map, versions, datasets)
  • FileHandler: class, processing one set of similar ROOT files
  • ObjectHandler: base abstract class, processing a comparison of one set of similar ROOT objects
  • Hist1DHandler: implementation of the ObjectHandler class for TH1 histograms
  • Hist2DHandler: implementation of the ObjectHandler class for TH2 histograms
  • Profile1DHandler: implementation of the OBjectHandler class for TProfile objects
Framework running macro: macro/qa/qa_compare.C

The macro processes comparison of files and objects defined in the objects.yaml (configName) configuration file for a selected setup (datasetName), obtained in two versions (placed in oldVersionInputDir and newVersionInputDir). The results of the comparison are saved into defined ROOT file (outputName).

Configuration YAML file contains a file-to-object names map, stored under the branch checker/files. Each file node contain three keys: "name" - name of the file, "label" - short label of the file and "objects" - array of object names. The "name" key contains a format of input file name, in which one should define format specifiers "%v" for a version path and "%d" for a dataset path. For example, for version path "/data/new" and dataset "s100e" the file name "%v/%d_qa.qa.root" will be interpreted as "/data/new/s100e_qa.qa.root". The "label" key is used as a name of the corresponding directory in output file. The "objects" key contains an array of file names stored in the input ROOT file. Please note, that the full path of the object inside the file should be provided.

Also, the configuration file can define the version names in a branch "checker/versions", the dataset names in a branch "checker/datasets" and the name of default version (branch "checker/default_label"). Each node of the versions branch accepts two keys for each version - "label" as a name of the version and "path" as a path to be used in input filename (see paragraph above). If the default version is not provided, the first defined version will be used as the default. Please note, that the versions and datasets should be defined only once either in a macro with getters or in the configuration file.

CI-test for QA-Checker

The CI test is defined in macro/qa/CMakeLists.txt. At the moment it executes the qa_compare.C macro with the same files treated as a new and old versions and created with the CI-tests of the run_qa.C macro.

Merge request reports