Add notification channel support in Slack #51
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Refactor the templates for PagerDuty and Slack to combine common elements into a single file, and update the Slack and PagerDuty receivers to use less local templating and use more defined templates. Add improved link generation for Alertmanager, including: - Improve the filter generation for the Alertmanger link to ensure that silenced Alerts are included in the search; - Create a new "all" silence link for Alertmanager which just uses the labels used for grouping as the filter, allowing easy access to silencing all related alerts without information about the metrics endpoint, Prometheus, and the alert configuration being included; and - Update the normal slience link to focus on only adding specific labels, if they exist, to the filter, such as information about the Pod or container, Node, and severity as so to make it less likly that the silence override will fail if there are changes to the metrics or Prometheus. Additionally, expand the possible links in the Slack message, using some of these new links and improve the naming and messaging around them too.
Add support for a new notifications chennel in the Slack provider configuration for Alertmanager which allows some Alerts to be sent there where they do not need to trigger an action directly, but may mean something needs adjusting. This can include information about Pods exceeding the memory or CPU requests as they are not exceeding the limits, and so won't be throttled or terminated. However, typical usage is now above the guaranteed amount, and the amount considered by the by the scheduler if the Pod is moved to another host, creating the potential for unstable operation.
Fix the initial sum_over_time() time-range for the PrometheusTSDBSeriesIncreaseNotice Alert as this was for three hours when the alert should be monitoring over an hours window for three hours.
Update some Prometheus and ElasticSearch alerts to use the new notifications channel in Slack to report on things happening but which are not directly an Alert, but useful information or signed that something may need to be fixed soon.
Fix the accidental setting of duplicate keys for cainjector in the cert-manager values file.
Fix the PodNotReady check to exclude the Succeeded phase for the Pods as this is a valid and health state and does not need to be alerted on.
jonathanio
added
priority/normal
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Add support to Alertmanager for a new
notificationschannel alongside theinfraandapplicationsalerts channels. Selected Alerts, such asPodspassing memory or CPU requests or other unusual or informational alerts that should probably be actioned but do not directly affect normal operational stability, are now redirected there.Checklist
Please check and confirm the following items have been performed, where
possible, for this Pull Request:
type/...,changes/..., and 'release/...' labels, as needed.