Skip to content

[CDASH] set hard limit for weekly tests timeouts on Virgo

Pierre-Alain Loizeau requested to merge p.-a.loizeau/cbmroot:limit_run_weekly_max_to into master

Try to fit the runtime of the weekly tests so that they can run in the main partition on Virgo by limiting the "obvious" too long timeouts.

These 4 places were leading by the automatic calculation to not realistic timeouts of 10000-20000 s for single tests (the main queue in Virgo is 8 hours or 28800 s for the full job of 350 tests, so too short even with 10 CPUs booked).

It is not 100% sure if this will be enough to make the automatic weekly upload to CDASH as apart from 2 or 3 tests most of them were still in the range of 1500 s duration and would therefore not hit the new limit.

Not sure whom should review this so I pinged the main actors under.

@s.zharko Does the QA ts_eb_ideal_compare test really need a fixed value of 10000 s or is it a typo?
from what I can see in my logs it typically finishes in 3-10 s even for the weekly tests

@f.kornas @f.uhlig For the JSON production test, is 40 events still reasonable?
or could it be that there is some part of the test which may conflict with other tests for disk access or such? \

  • On Virgo I see (in the last few weeks) the test passing most of the time after ~1200-1400 s but sometimes timing out after more than 6000 s
  • on the mcbmop nodes of the control room, it similarly oscillates between 2500 s and more than 6000 s depending on the node and the week

Merge request reports