scheduler.py: Try to not fail if submit have exception #367
Conversation
We are having quite often errors on LAVA submit, and this will cause scheduler to crash. This is generic attempt to keep running on such failures, which might be handled better in future. Signed-off-by: Denys Fedoryshchenko <denys.f@collabora.com>
| try: | ||
| running_job = runtime.submit(output_file) | ||
| except Exception as e: | ||
| self.log.error(' '.join([ |
There was a problem hiding this comment.
It would probably be neater to collect node/runtime/job data first and append it with TestJob ID/error message for logging but it could be the next step
There was a problem hiding this comment.
yes, and any failure of schedulers or any processes in pipeline very big and interesting topic, where we need to retry, where we need to log failure, and how to log it.
yesterday at 11:39:02 PM12/07/2023 09:39:02 PM UTC [ERROR] 65723b76ea59c4c039f324ff lava-collabora-staging minnowboard-turbot-E3826 baseline-x86 400 Client Error: Bad Request for url: https://staging.lava.collabora.dev/api/v0.2/jobs/?format=json&limit=256 |
We are having quite often errors on LAVA submit, and this will cause scheduler to crash.
This is generic attempt to keep running on such failures, which might be handled better in future.