Bump flesnet

I will try to revive my branch cleaning out the old fles/mcbm2018 folder tomorrow or on Friday once I am done with administrative things. That should remove at least one of the need for upgrade

I suspect that the build failes because of an issue reported earlier by @f.uhlig, which is that Boost in fairsoft is built in a way that excludes zstd.

@f.uhlig, what is your take – how should we proceeed?

I had a look at the dependencies doc of the Apr22 and Nov22 releases and see only the following compression packages:

bzip2
gzip
libbz2-dev

I did not find any explicit disabling of zstd in the Fairsoft source of these two versions

So most probably the majority of our system do not have the package installed and then boost auto-disable this feature at configuration, while if present it would be enabled also automatically (ROOT brings its own zstd when it detects that it is not present in the OS... probably why we did not think about it before).

Just thinking on how we could enable quickly Norbert to have a look at least at the decompressed version of the TSA files, would it make sense/be possible to add a few commits to Flesnet to

Detect in the cmake config stage if zstd is present in the system and set a cmake variable to store the result
Keep the ArchiveDescriptor as is
Surround the zstd include and part of the (de)compression blocks in InputArchive, OutputArchive and OutputArchiveSequence with some preprocessor switches, with an automatic Unsupported Compression exception if zstd is not present and the descriptor flag is not none

I would expect that then the compilation may go through, the compressed files would still fail and the decompressed files would go through

(Apart from the detection step, I would probably manage myself to do this by hand in a copy of cbmroot and check if it works before makeing a PR to Flesnet)

@j.decuveland @n.herrmann The following patch to the flesnet version of this MR allowed compilation in my tests on Virgo, properly failed on the compressed file and went through on the uncompressed one.
I will test tomorrow on cbmfles01 to make sure that the library is still properly detected when compiling on a node where it is present in the OS side.

I however got an error from the TOF unpacker on the uncompressed file prepared by Norbert, so maybe still something fishy (ProcessEpoch => Error first global epoch, with a diff of +1)

diff --git a/lib/fles_ipc/CMakeLists.txt b/lib/fles_ipc/CMakeLists.txt
index f335224e..cef5755a 100644
--- a/lib/fles_ipc/CMakeLists.txt
+++ b/lib/fles_ipc/CMakeLists.txt
@@ -4,6 +4,13 @@
 file(GLOB LIB_SOURCES *.cpp)
 file(GLOB LIB_HEADERS *.hpp)
 
+if(NOT ZSTD_FOUND)
+  message(STATUS "ZSTD not found.")
+else()
+  message(STATUS "ZSTD found.")
+  add_compile_definitions(WITH_ZSTD)
+endif()
+
 add_library(fles_ipc ${LIB_SOURCES} ${LIB_HEADERS})
 
 target_compile_definitions(fles_ipc PUBLIC BOOST_ALL_DYN_LINK)
diff --git a/lib/fles_ipc/InputArchive.hpp b/lib/fles_ipc/InputArchive.hpp
index 91fc8492..b2eb2a9f 100644
--- a/lib/fles_ipc/InputArchive.hpp
+++ b/lib/fles_ipc/InputArchive.hpp
@@ -6,7 +6,9 @@
 #include "ArchiveDescriptor.hpp"
 #include "Source.hpp"
 #include <boost/archive/binary_iarchive.hpp>
-#include <boost/iostreams/filter/zstd.hpp>
+#ifdef WITH_ZSTD
+  #include <boost/iostreams/filter/zstd.hpp>
+#endif
 #include <boost/iostreams/filtering_stream.hpp>
 #include <fstream>
 #include <memory>
@@ -43,6 +45,7 @@ public:
     }
 
     if (descriptor_.archive_compression() != ArchiveCompression::None) {
+#ifdef WITH_ZSTD
       in_ = std::make_unique<boost::iostreams::filtering_istream>();
       if (descriptor_.archive_compression() == ArchiveCompression::Zstd) {
         in_->push(boost::iostreams::zstd_decompressor());
@@ -54,6 +57,11 @@ public:
       in_->push(*ifstream_);
       iarchive_ = std::make_unique<boost::archive::binary_iarchive>(
           *in_, boost::archive::no_header);
+#else
+      throw std::runtime_error(
+          "Unsupported compression type for input archive file \"" +
+          filename + "\"");
+#endif
     }
   }
 
diff --git a/lib/fles_ipc/OutputArchive.hpp b/lib/fles_ipc/OutputArchive.hpp
index 4e480ac1..6787d804 100644
--- a/lib/fles_ipc/OutputArchive.hpp
+++ b/lib/fles_ipc/OutputArchive.hpp
@@ -6,7 +6,9 @@
 #include "ArchiveDescriptor.hpp"
 #include "Sink.hpp"
 #include <boost/archive/binary_oarchive.hpp>
-#include <boost/iostreams/filter/zstd.hpp>
+#ifdef WITH_ZSTD
+  #include <boost/iostreams/filter/zstd.hpp>
+#endif
 #include <boost/iostreams/filtering_stream.hpp>
 #include <fstream>
 #include <string>
@@ -37,6 +39,7 @@ public:
     *oarchive_ << descriptor_;
 
     if (compression != ArchiveCompression::None) {
+#ifdef WITH_ZSTD
       out_ = std::make_unique<boost::iostreams::filtering_ostream>();
       if (compression == ArchiveCompression::Zstd) {
         out_->push(boost::iostreams::zstd_compressor(
@@ -49,6 +52,11 @@ public:
       out_->push(ofstream_);
       oarchive_ = std::make_unique<boost::archive::binary_oarchive>(
           *out_, boost::archive::no_header);
+#else
+      throw std::runtime_error(
+          "Unsupported compression type for output archive file \"" +
+          filename + "\"");
+#endif
     }
   }
 
diff --git a/lib/fles_ipc/OutputArchiveSequence.hpp b/lib/fles_ipc/OutputArchiveSequence.hpp
index 47b8e022..f1df2e63 100644
--- a/lib/fles_ipc/OutputArchiveSequence.hpp
+++ b/lib/fles_ipc/OutputArchiveSequence.hpp
@@ -7,7 +7,9 @@
 #include "Sink.hpp"
 #include <boost/algorithm/string.hpp>
 #include <boost/archive/binary_oarchive.hpp>
-#include <boost/iostreams/filter/zstd.hpp>
+#ifdef WITH_ZSTD
+  #include <boost/iostreams/filter/zstd.hpp>
+#endif
 #include <boost/iostreams/filtering_stream.hpp>
 #include <cstdint>
 #include <fstream>
@@ -129,6 +131,7 @@ private:
     *oarchive_ << descriptor_;
 
     if (descriptor_.archive_compression() != ArchiveCompression::None) {
+#ifdef WITH_ZSTD
       out_ = std::make_unique<boost::iostreams::filtering_ostream>();
       if (descriptor_.archive_compression() == ArchiveCompression::Zstd) {
         out_->push(boost::iostreams::zstd_compressor(
@@ -141,6 +144,11 @@ private:
       out_->push(*ofstream_);
       oarchive_ = std::make_unique<boost::archive::binary_oarchive>(
           *out_, boost::archive::no_header);
+#else
+      throw std::runtime_error(
+          "Unsupported compression type for output archive file \"" +
+          filename(file_count_) + "\"");
+#endif
     }
 
     ++file_count_;

After a more careful look at the Flesnet main CMake file, I now see that the dependency on ZSTD is checked only on Apple. So what would be needed instead is a way to check in CMake if boost was compiled/installed with ZSTD support.

Observations after testing the building on different computers:

with libzstd-dev installed in the OS, both ROOT and BOOST are automatically compiled in Fairsoft with Zstd support
in this case the simple Flesnet hash update in Cbmroot should work (ldd on the libboost_iostreams library shows the dependency on libzstd), but it still crashes wit the same error message at the linking stage (no error message in the compilation of the flesnet target in Cbmroot)
the linking error seems to come when the libfles_ipc.a we build for Cbmroot is added to one of the Cbmroot binaries, so I am not fully sure if the problem is really on the boost in Fairsoft and not more on something wrong in our dependencies declaration (either in the Cmake files of fles_ipc itself or in our InstallFlesnet.cmake)
in addition my patch does not works to enable the ZSTD support as it detects the presence of libzstd-dev, not whether boost was compiled with it or not, which make troubles if one has FS versions compiled before and after the installation of the package (and the detection is anyway run only on APPLE in flesnet)
the new requirement of CMake 3.22 for Flesnet will fail some of our test systems (Debian 10 w/ 3.18) as the latest Fairsoft provides only 3.16.1 (so it will work only only if the OS anyway provides a higher version)

Unfortunately the first entries on google do not really explain how to properly setup the usage of boost iostream + the compression filters, especially with CMake. There are a few from the MS Win world with similar output which seem to have been resolved bu changing the order in which external libraries are passed to the linker with -lxxxx.
So from my side I am for now a bit stuck and do not see how to fix it so that it passes all our test systems while allowing to read the compressed files.
Maybe @d.hutter has more insights in the linking problem from working on it from the other side adding the dependency into the fles_ipc lib instead of looking down trying to include fles_ipc as I did.

I prepared an MR toward Jan branch to apply my patch from the Cbmroot side as a temporary solution.

you can find it at j.decuveland/cbmroot!2 (closed)

I am currently trying to get the development packages of zstd and lzma into our Debian10 container (vae23). This would allow to recompile the FairSoft boost installation and should allow to use the latest flesnet version with CbmRoot (at least with Debian 10 on the Virgo cluster)

@f.uhlig @j.decuveland I found the reason for the linker error still being there when zstd is present: the iostreams component was not required in the Cbmroot call to Find_Boost, so the corresponding cmake variables were undefined/empty.
=> I will add this to my MR to Jan branch.

With this we should be able to support 2 out of the 3 possible cases:

If not zstd-dev present, my patch make it so that only the uncompressed files are supported
If zstd-dev is present and was there before Fairsoft (and boost) were compiled, both uncompressed and compressed files are supported
If zstd-dev is present but was not there before Fairsoft (and boost) were compiled, both files type will be "supported" but either linking or usage of compressed files may fail

Edit: I tested the two first cases and they behave as I thought, may try the third one later this afternoon or tomorrow after finding a proper computer

Impressive. P-A, I thought very similar things to what you reported three days ago, but did not yet get anywhere close to your level of implementing it. Compiling with the current flesnet code but without compression support would also be my idea for a workaround. The whole thing is quite complicated indeed. I'll briefly try one other thing though...

Btw, also as a remark here: As a temporary workaround, I used a patched version of flesnet to create extracted files named *.legacy.tsa, which should be compatible with the old cbmroot/flesnet versions.

@j.decuveland

With @f.uhlig help I could find a way to test if the boost::iostream library has zstd filtering support. I updated my MR to your branch with it.
=> With this I could check on my test computers that in all 3 cases we get consistent behavior (meaningful exception if uncompression not available in boost, processing otherwise) with all 3 files types (legacy work always, uncompressed work always, compressed work only in case 2)

I think it could also easily be extended to similarly detect LZMA or BZIP support, so maybe we may want to apply my first commit directly to flesnet instead of patching on the cbmroot side as a temporary step.
The second commit I think will still be needed to make Cbmroot compatible with the new Flesnet

File opening works now! Thanks a lot!

[RepReqTsSampler][12:04:41.157437][INFO][CbmMQTsSamplerRepReq.cxx:225:InitTask] Input Directory: /lustre/cbm/prod/beamtime/2023/12/mcbm [2024-01-17 12:04:41.203818] [0x00007fe1041dcec0] [info] Number of input streams: 1 [2024-01-17 12:04:41.203865] [0x00007fe1041dcec0] [info] Number of files in Stream: 1 [2024-01-17 12:04:41.203870] [0x00007fe1041dcec0] [info] File: /lustre/cbm/prod/beamtime/2023/12/mcbm/2664_node8_00_0000.legacy.tsa [2024-01-17 12:04:41.271087] [0x00007fe1041dcec0] [info] Open file: /lustre/cbm/prod/beamtime/2023/12/mcbm/2664_node8_00_0000.legacy.tsa [RepReqTsSampler][12:04:41.281541][INFO][CbmMQTsSamplerRepReq.cxx:230:InitTask] High-Water Mark: 10 [RepReqTsSampler][12:04:41.281564][INFO][CbmMQTsSamplerRepReq.cxx:231:InitTask] Max. Timeslices: 18446744073709551615 [RepReqTsSampler][12:04:41.281574][INFO][CbmMQTsSamplerRepReq.cxx:232:InitTask] Sending TS copies in no-split mode [RepReqTsSampler][12:04:41.281629][STATE][StateMachine.cxx:194:ProcessWork] INITIALIZING TASK ---> READY [RepReqTsSampler][12:04:41.281706][STATE][StateMachine.cxx:194:ProcessWork] READY ---> RUNNING [RepReqTsSampler][12:04:41.281723][INFO][Device.cxx:438:RunWrapper] fair::mq::Device running...

Unpacking, however, is crashing: [12:05:16][INFO] CbmTofUnpackAlgo::initParSetTofMcbm2018 - Successfully initialized TOF settings [12:05:16][INFO] CbmTofUnpackAlgo::SystemTimeOffset:0 [12:05:16][INFO] Timeslice parameters: each TS has 100 Core MS and 1 Overlap MS, for a MS duration of 1.28e+06 ns, a core duration of 1.28e+08 ns and a full duration of 1.2928e+08 ns [12:05:16][ERROR] CbmTofUnpackAlgo::ProcessEpoch => Error first global epoch, from MS index 0xacce62, current 0xacce63, diff 1, raw 0x0000ff00acce6301, NoErr 0, current 0x3c80d3acce62 66524004798050.0000 00 [12:05:16][FATAL] CbmTofUnpackAlgo::ProcessEpoch => Stopping there, system is not synchronized and send corrupt data

Any suggestion would be welcome.