Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve grafana configuration steps #5795

Merged
merged 6 commits into from
Jan 5, 2024
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 50 additions & 13 deletions docs/snippets/collecting-metrics.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,17 @@ You can also set up the OpenTelemetry Collector to receive metrics from Knative
!!! warning
You can't use OpenTelemetry Collector and Prometheus at the same time. The default metrics backend is Prometheus. You will need to remove `metrics.backend-destination` and `metrics.request-metrics-backend-destination` keys from the config-observability Configmap to enable Prometheus metrics.

## About Prometheus
## About the Prometheus Stack

[Prometheus](https://prometheus.io/) is an open-source tool for collecting,
aggregating timeseries metrics and alerting. It can also be used to scrape the OpenTelemetry Collector that is demonstrated below when Prometheus is used.
[Prometheus](https://prometheus.io/) is an open-source tool for collecting, aggregating timeseries metrics and alerting. It can also be used to scrape the OpenTelemetry Collector that is demonstrated below when Prometheus is used.

## Setting up Prometheus
[Grafana](https://grafana.com/oss/) is an open-source platform for data analytics and visualization, enabling users to create customizable dashboards for monitoring and analyzing metrics from various data sources.

1. Install the [Prometheus Operator](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) by using [Helm](https://helm.sh/docs/intro/using_helm/):
[Prometheus Stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) is a preconfigured collection of Kubernetes manifests, Grafana dashboards, and Prometheus rules, combined to provide end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. The stack includes by default some Prometheus packages and Grafana.

## Setting up the Prometheus Stack

1. Install the [Prometheus Stack](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) by using [Helm](https://helm.sh/docs/intro/using_helm/):

```bash
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
Expand Down Expand Up @@ -47,27 +50,61 @@ aggregating timeseries metrics and alerting. It can also be used to scrape the O
kubectl apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/servicemonitor.yaml
```

1. Grafana dashboards can be imported from the [`knative-extensions` repository](https://github.com/knative-extensions/monitoring/tree/main/grafana).
### Access the Prometheus instance locally

1. If you are using the Grafana Helm Chart with the Dashboard Sidecar enabled, you can load the dashboards by applying the following configmaps.
By default, the Prometheus instance is only exposed on a private service named `prometheus-kube-prometheus-prometheus`.

To access the console in your web browser:

1. Enter the command:

```bash
kubectl apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/grafana/dashboards.yaml
kubectl port-forward -n default svc/prometheus-kube-prometheus-prometheus 9090:9090
```

### Access the Prometheus instance locally
1. Access the console in your browser via `http://localhost:9090`.

By default, the Prometheus instance is only exposed on a private service named `prometheus-operated`.
### Access the Grafana instance locally

To access the console in your web browser:
By default, the Grafana instance is only exposed on a private service named `prometheus-grafana`.

To access the dashboards in your web browser:

1. Enter the command:

```bash
kubectl port-forward -n default svc/prometheus-operated 9090
kubectl port-forward -n default svc/prometheus-grafana 3000:80
```

1. Access the console in your browser via `http://localhost:9090`.
1. Access the dashboards in your browser via `http://localhost:3000`.

1. Use the default credentials to login:

```text
username: admin
password: prom-operator
```

### Import Grafana dashboards

1. Grafana dashboards can be imported from the [`monitoring` repository](https://github.com/knative-extensions/monitoring/tree/main/grafana).

1. If you are using the Grafana Helm Chart with the Dashboard Sidecar enabled, you can load the dashboards by applying the following configmaps.

```bash
kubectl apply -f https://raw.githubusercontent.com/knative-extensions/monitoring/main/grafana/dashboards.yaml
```

!!! caution
You will need to ensure that the helm chart has following values configured, otherwise the dashboards loading will not work.
```yaml
grafana:
sidecar:
dashboards:
enabled: true
searchNamespace: ALL
```
If you have an existing configmap and the dashboards loading doesn't work, add the `labelValue: true` attribute to the helm chart after the `searchNamespace: ALL` declaration.

## About OpenTelemetry

Expand Down
Loading