Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vmcluster: improve components readiness check #1907

Merged
merged 8 commits into from
Jan 9, 2025

Conversation

func25
Copy link
Contributor

@func25 func25 commented Jan 8, 2025

  • Remove vmstorage liveness probe, as vminsert already handles routing and retries, while liveness probes can inadvertently introduce delays, DNS instability, and unnecessary disruptions.
  • Reduce the default readiness probe interval to 5s (was 15s) and the failure threshold to 10 (was 3).
  • Add a default minReadySeconds for vmstorage, to help stabilizing service during rollout.

@github-actions github-actions bot added cluster vmcluster helm chart related issue metrics labels Jan 8, 2025
@Haleygo
Copy link
Contributor

Haleygo commented Jan 8, 2025

Could you also help adjust the values for other components like vminsert, vmselect, and vmauth? 15s seems too long for all of them.

probe:
# -- VMSelect readiness probe
readiness:
httpGet: {}
initialDelaySeconds: 5
periodSeconds: 15
timeoutSeconds: 5
failureThreshold: 3

# -- Readiness & Liveness probes
probe:
# -- VMInsert readiness probe
readiness:
httpGet: {}
initialDelaySeconds: 5
periodSeconds: 15
timeoutSeconds: 5
failureThreshold: 3

@func25
Copy link
Contributor Author

func25 commented Jan 8, 2025

@Haleygo, also updated the vmbackupmanager. LMK if you want to revert this change.

@AndrewChubatiuk AndrewChubatiuk force-pushed the vmcluster-probe-enhance branch from 6d7e769 to 91e88e8 Compare January 8, 2025 07:38
@Haleygo Haleygo changed the title vmcluster: enhance vmstorage readiness probe vmcluster: improve components readiness check Jan 8, 2025
Co-authored-by: Phuong Le <39565248+func25@users.noreply.github.com>
@daniel-lee-sb
Copy link

This is really needed 👍 Thanks for the quick update.

Haleygo
Haleygo previously approved these changes Jan 9, 2025
@Haleygo
Copy link
Contributor

Haleygo commented Jan 9, 2025

cc @AndrewChubatiuk

@Haleygo Haleygo force-pushed the vmcluster-probe-enhance branch from 20e16a2 to 2b11f66 Compare January 9, 2025 06:29
@Haleygo Haleygo merged commit 8e64912 into master Jan 9, 2025
8 checks passed
@Haleygo Haleygo deleted the vmcluster-probe-enhance branch January 9, 2025 09:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cluster vmcluster helm chart related issue metrics
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants