Distinguish connection errors from "real" errors in nightly builds #422

andrewmogan · 2025-01-09T17:17:50Z

Over the past few months, we've had transient failures in the nightly builds due to spack FetchErrors. This indicates a simple connection error where the solution is to manually rerun the build until the connection is resolved. This issue is not reproducible since it occurs sporadically and does not necessarily affect the same package every time. When this occurs, we could automatically rerun the build by scanning the GitHub Actions output for a FetchError and returning a specific exit code in our workflow which can then be passed to this external retry action to automatically rerun the failed step.

The text was updated successfully, but these errors were encountered:

… action

…ilure

…ack-repo directory

…r full install after retry

…instead of a feature branch

… reporting; alsa opportunistically drop dbe-build logic from the v4 nightly build since dbe isn't a thing in v4

andrewmogan · 2025-01-24T21:58:58Z

Addressed by #423.

andrewmogan added a commit that referenced this issue Jan 9, 2025

Issue #422: Use retry action on coredaq build step of v5 nightly

69b64e0

andrewmogan added a commit that referenced this issue Jan 9, 2025

Issue #422: Arbitrarily set (required) retry timeout

1e90ec5

andrewmogan added a commit that referenced this issue Jan 9, 2025

Issue #422: Checkout this feature branch of daq-release for testing

384eaa9

andrewmogan added a commit that referenced this issue Jan 9, 2025

Issue #422: Close if statements

b8639fc

andrewmogan added a commit that referenced this issue Jan 9, 2025

Issue #422: Add file argument to grep

8ac9b30

andrewmogan added a commit that referenced this issue Jan 9, 2025

Issue #422: Specify bash shell in retry action

9c957aa

andrewmogan added a commit that referenced this issue Jan 9, 2025

Issue #422: Add debug echo statements

51f530b

andrewmogan added a commit that referenced this issue Jan 9, 2025

Issue #422: Add echo debug statements

2757d16

andrewmogan added a commit that referenced this issue Jan 9, 2025

Issue #422: Propagate exit 111 to github actions step

9025f33

andrewmogan added a commit that referenced this issue Jan 9, 2025

Issue #422: Set exist_ok to True for generate_daq_package

47519b6

andrewmogan added a commit that referenced this issue Jan 9, 2025

Issue #422: Set exist_ok to True everywhere

35fb1f3

andrewmogan added a commit that referenced this issue Jan 9, 2025

Issue #422: Revert exist_ok=True's and set on_retry_command for retry…

7c1c268

… action

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Sync coredaq and fddaq install steps

041feec

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Remove debug statements

32eedfa

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Reimplement full spack install

63520d1

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Replace retry action with manual logic

aafc30b

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: echo spack install exit code

1d1bb29

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Cleanup directories before installing

7e4fc91

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Attempt to spack rm a problematic directory

5b2e420

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Move spack rm command

d0c86df

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Move spack repo rm to build-release.sh

9960717

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Fix release manifest path

781d0eb

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Use exit 111 for simulated failure

41672cd

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Revert to previous exit code and debug statements

dd01c11

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Clean build directories before spack install

9fc8fce

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Fix build-release argument

3090755

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Move and narrow cleanup commands

d5adf78

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Remove spack-repo directory on exit 111

690411d

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Remove spack default directory on retry

c122ea8

andrewmogan added a commit that referenced this issue Jan 10, 2025

Issue #422: Move rm commands to workflow rather than build-release.sh

9063ecb

andrewmogan added a commit that referenced this issue Jan 14, 2025

Issue #422: Keep spack version directory and revert to intentional fa…

f70eaa4

…ilure

andrewmogan added a commit that referenced this issue Jan 14, 2025

Issue #422: Revert to full spack install on first retry

b270257

andrewmogan added a commit that referenced this issue Jan 14, 2025

Issue #422: Revert back to intentional failure on retry and remove sp…

9417060

…ack-repo directory

andrewmogan added a commit that referenced this issue Jan 14, 2025

Issue #422: Revert back to full install on first retry

f1557e7

andrewmogan added a commit that referenced this issue Jan 14, 2025

Issue #422: Remove deprecated if statement

521608a

andrewmogan added a commit that referenced this issue Jan 14, 2025

Issue #422: ls SPACK_AREA for debugging

787bc35

andrewmogan added a commit that referenced this issue Jan 14, 2025

Issue #422: ls deeper for more debugging

2b2e806

andrewmogan added a commit that referenced this issue Jan 14, 2025

Issue #422: More ls

3060071

andrewmogan added a commit that referenced this issue Jan 14, 2025

Issue #422: Remove more specific spack directories and attempt anothe…

677e601

…r full install after retry

andrewmogan added a commit that referenced this issue Jan 14, 2025

Issue #422: Add more ls and recursive diff for debugging

aaddcc0

andrewmogan added a commit that referenced this issue Jan 14, 2025

Issue #422: Don't rm anything, see what happens

1899ee7

andrewmogan added a commit that referenced this issue Jan 14, 2025

Issue #422: Fix coredaq artifact upload arguments

0f75a3b

andrewmogan added a commit that referenced this issue Jan 14, 2025

Issue #422: Intentionally fail each retry

a080143

andrewmogan added a commit that referenced this issue Jan 15, 2025

Issue #422: Run full spack install each time, but still force failure

73f2ab6

andrewmogan added a commit that referenced this issue Jan 15, 2025

Issue #422: Fail the first two builds, then succeed

9438a2d

andrewmogan added a commit that referenced this issue Jan 15, 2025

Issue #422: Clean up: remove debug statements, comments, etc.

23343e5

andrewmogan added a commit that referenced this issue Jan 15, 2025

Issue #422: Add retry logic with intentional failure to v4 build

4b4bf17

andrewmogan added a commit that referenced this issue Jan 15, 2025

Issue #422: Remove intentional failures

956c3db

andrewmogan added a commit that referenced this issue Jan 15, 2025

Issue #422: Prepare for merge by checking out develop of daq-release …

c222a4e

…instead of a feature branch

andrewmogan mentioned this issue Jan 15, 2025

Automatically rerun nightly builds which fail due to transient connection errors #423

Merged

andrewmogan added a commit that referenced this issue Jan 23, 2025

Issue #422: Extend retry logic to dbe

8d493a4

andrewmogan added a commit that referenced this issue Jan 23, 2025

Issue #422: Intentionally fail every build attempt

e522798

andrewmogan added a commit that referenced this issue Jan 24, 2025

Issue #422: Make sure fddaq step doesn't continue on failure

d7ce12e

andrewmogan added a commit that referenced this issue Jan 24, 2025

Issue #422: Remove intentional failure

5c5e726

andrewmogan added a commit that referenced this issue Jan 24, 2025

Issue #422: Sync retry logic for v4 build

362b03b

andrewmogan added a commit that referenced this issue Jan 24, 2025

Issue #422: Simulate non-retryable exit code on fddaq step

b96f055

andrewmogan added a commit that referenced this issue Jan 24, 2025

Issue #422: Revert intentional failures

a13be1e

jcfreeman2 pushed a commit that referenced this issue Jan 24, 2025

JCF: Issue #422: add attempt reporting to dbe alongside coredaq/fddaq…

13b9dd0

… reporting; alsa opportunistically drop dbe-build logic from the v4 nightly build since dbe isn't a thing in v4

andrewmogan closed this as completed Jan 24, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Distinguish connection errors from "real" errors in nightly builds #422

Distinguish connection errors from "real" errors in nightly builds #422

andrewmogan commented Jan 9, 2025

andrewmogan commented Jan 24, 2025

Distinguish connection errors from "real" errors in nightly builds #422

Distinguish connection errors from "real" errors in nightly builds #422

Comments

andrewmogan commented Jan 9, 2025

andrewmogan commented Jan 24, 2025