Add retry logic to sample app endpoint connection #625
Conversation
Codecov ReportAttention:
❗ Your organization needs to install the Codecov GitHub app to enable full functionality. Additional details and impacted files@@ Coverage Diff @@
## main #625 +/- ##
=============================================
- Coverage 85.71% 50.73% -34.99%
- Complexity 19 264 +245
=============================================
Files 3 39 +36
Lines 49 1301 +1252
Branches 5 141 +136
=============================================
+ Hits 42 660 +618
- Misses 3 609 +606
- Partials 4 32 +28 ☔ View full report in Codecov by Sentry. |
| with: | ||
| max_attempts: 3 | ||
| retry_on: error | ||
| timeout_minutes: 6 |
There was a problem hiding this comment.
What is this timeout for? Is it the time limit for the all the commands to run successfully before retrying?
There was a problem hiding this comment.
Yup it's the time limit for all the commands to run. On a successful run it takes less than 2 minutes, but on failed runs it seems like it takes ~6 minutes.
| shell: bash | ||
| # Restart the pod, then attempt to connect to the endpoint. It will try 30 times with 10 seconds delay. If | ||
| # it fails to establish connection, restart the pod and try again up to 3 tries. | ||
| command: | |
There was a problem hiding this comment.
I'm a little confused as to what is the root cause of the transient failure this PR is aiming to fix. Is it that the application pods are occasionally fail to run or is it the ALB endpoint that fails to come up. I feel its the latter.
In that case, does simply restarting the application pods solves the ALB endpoint issue? Have we reproduced/tested this fix anywhere?
There was a problem hiding this comment.
Yes, the failure happens when the endpoints fails to come up. We think it's happening due to occasional network problem/timeouts, so in such situations restarting the pod should hopefully allow the endpoints to come up properly. It's hard to reproduce this error, so I'm not 100% sure if it will work, but tested the code to check if the entire workflow succeeds properly and tested to see if the retry logic runs properly in the event that the commands fail.
|
Switched out of the |
| retry_counter=0 | ||
| max_retry=3 | ||
| success=0 | ||
| while [ $success -eq 0 ]; do | ||
| if [ $retry_counter -ge $max_retry ]; then | ||
| exit 1 | ||
| fi | ||
| kubectl delete pods --all -n ${{ env.SAMPLE_APP_NAMESPACE }} | ||
| kubectl wait --for=condition=Ready pod --all -n ${{ env.SAMPLE_APP_NAMESPACE }} | ||
| sample_app_endpoint=http://$(terraform output sample_app_endpoint) | ||
| if ! $(curl --retry 30 --retry-delay 10 -s -o /dev/null $(echo "$sample_app_endpoint" | tr -d '"'));then | ||
| echo "Failed to establish connection with endpoint" | ||
| retry_counter=$(($retry_counter+1)) | ||
| else | ||
| success=1 | ||
| fi | ||
| done |
There was a problem hiding this comment.
In the worst case, how long will this take? Can we justify taking that long for a single test?
Issue #, if available:
The E2E canary occasionally fails due to being unable to establish connection with the sample app.
Description of changes:
If connecting to the endpoint fails, then restart the sample app again and reattempt establishing connection; up to 3 times
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.