[CI] Add detection of job timeout and total test time TO check to CI
Two features:
- Add a script to create a ctest absolute time limit based on the GITLAB job timeout with a bit less than 5 minutes margin (depending on when in the job the script is executed)
- Add another script allowing to set an absolute limit in second for the total time of the test stage of the build, also relying in the ctest
stop time
. In case the above script is also used the earliest of the two times is used
=> Both scripts rely on the already existing infrastructure added to handle the limited duration of the SLURM jobs for the weekly builds on Virgo.
=> The total test time timeout is for now provided for the real_data
build
Redmine: refs #3462
Set as draft until sure all functionalities work, then a second commit should be added to tune the test phase total time timeout for the real_data
build to some reasonable start value (may also need to wait until the problem of the too long init time of tracking is resolved and the build times are back close to the original duration)
Merge request reports
Activity
requested review from @f.uhlig
assigned to @f.uhlig
- Resolved by Florian Uhlig
From the logs of the build job it seem that there are still errors in my changes to the CMAKE file:
- either
$CI_JOB_TIMEOUT
is defined inUTC
and not local time, or it is not set at all. -
CI_TESTS_TOTAL_END_TIME_DATE
is a cmake variable and not anENV
one at line 137
I will check/fix this when I am back next week.
- either
added 13 commits
-
2e1036a0...7beb79fd - 12 commits from branch
computing:master
- 917162d9 - [CI] Add detection of job timeout and total test time TO check to CI
-
2e1036a0...7beb79fd - 12 commits from branch
@f.uhlig For reference in case we want to change the pipelines timeout of alll our forks (I checked that my own fork has only
1h
while the central project has2h
): the keyword isbuild_timeout
, described in https://docs.gitlab.com/ee/api/projects.html, which takes an integer value in secondsadded 1 commit
- 8c25b507 - [CI] Add detection of job timeout and total test time TO check to CI
added 1 commit
- d3801b00 - [CI] Add detection of job timeout and total test time TO check to CI
added 1 commit
- b3becf6a - [CI] Add detection of job timeout and total test time TO check to CI
added 1 commit
- 13315e47 - [CI] Add job timeout specific to build stage jobs, set to 1h30 for now
@f.uhlig I found a way to circumvent the project/fork setting, as since Gitlab
15.7
there is a new CI YAML keyword which allow to set the timeout for a single job (can be longer than the project limit so long as shorter than the runner one). So this way the timeout would be shipped with the code and versioned ingit
.Still not working with the ctest limit as the resulting variable is not properly set it seem, so more draft/debugging needed
added 4 commits
Toggle commit listadded CI/CD label
Removed draft mode after:
- Cleanup up the printouts
- re-organizing the commits to highlight the main 3 changes
- Setting the
real_data
test TO to2100 s
because round number and +100-150 s larger than most MR in last 2 days (the MR fixing the material maps seem to bring it down to around1500 s
)
Dear @f.uhlig, @v.friese, @p.-a.loizeau,
you have been identified as code owner of at least one file which was changed with this merge request.
Please check the changes and approve them or request changes.
added CodeOwners label
added 11 commits
-
4f070414...75737e77 - 8 commits from branch
computing:master
- a77c624c - [CI] Add job timeout specific to build stage jobs, set to 1h30 for now
- d9c19c99 - [CI] Add detection of job timeout and total test time TO check to CI
- ae9ddeb5 - [CI] Set the total test timeout for real data to 35 minutes
Toggle commit list-
4f070414...75737e77 - 8 commits from branch
added 4 commits
- 6e1c02d3 - [CI] Add job timeout specific to build stage jobs, set to 1h30 for now
- 264800fd - [CI] Add detection of job timeout and total test time TO check to CI
- 0a27b0e0 - [CI] Set the total test timeout for real data to 35 minutes
- b4473b99 - [CI] Reduce job TO to 1h + test TO for real data to 25m now that !1952 (merged) is in
Toggle commit list