[CDASH] set hard limit for weekly tests timeouts on Virgo
Try to fit the runtime of the weekly tests so that they can run in the main partition on Virgo by limiting the "obvious" too long timeouts.
These 4 places were leading by the automatic calculation to not realistic timeouts of 10000-20000 s for single tests (the main queue in Virgo is 8 hours or 28800 s for the full job of 350 tests, so too short even with 10 CPUs booked).
It is not 100% sure if this will be enough to make the automatic weekly upload to CDASH as apart from 2 or 3 tests most of them were still in the range of 1500 s duration and would therefore not hit the new limit.
Not sure whom should review this so I pinged the main actors under.
@s.zharko Does the QA ts_eb_ideal_compare test really need a fixed value of 10000 s or is it a typo?
from what I can see in my logs it typically finishes in 3-10 s even for the weekly tests
@f.kornas @f.uhlig For the JSON production test, is 40 events still reasonable?
or could it be that there is some part of the test which may conflict with other tests for disk access or such? \
- On
VirgoI see (in the last few weeks) the test passing most of the time after~1200-1400 sbut sometimes timing out after more than6000 s - on the
mcbmopnodes of the control room, it similarly oscillates between2500 sand more than6000 sdepending on the node and the week