Slides here
- Prometheus: http://localhost:9090
- Grafana: http://localhost:3000 (user: admin / pass: workshop)
- Node-Exporter: http://localhost:9100/metrics
- PostgreSQL (2 tables): postgres://devfest:devfest@localhost:5432/devfest
- Postgresql-Exporter: http://localhost:9187/metrics
- Nginx: http://localhost:8080
- Nginx-Exporter: http://localhost:9101/metrics
- Springboot-exmaple-Exporter: http://127.0.0.1:8081/actuator/prometheus
- Python-exmaple-Exporter: http://127.0.0.1:5000/metrics
Take a look on Prometheus metric types (counter, gauges, histogram, summary) => https://prometheus.io/docs/concepts/metric_types/
# Starts Prometheus
docker-compose up -d prometheus
# Starts system metrics exporter
docker-compose up -d node-exporter
- Prometheus console: http://localhost:9090.
- Full list of ingested metrics: http://localhost:9090/graph.
node-exportermetrics: http://localhost:9100/metrics.
Update prometheus.yml config file, to scrape node-exporter metrics every 10 seconds. 🚀
💡 Solution
#
# /etc/prometheus/prometheus.yml
#
global:
# default scrape_interval
scrape_interval: 30s
scrape_configs:
- job_name: 'node-exporter'
scrape_interval: 10s
static_configs:
- targets: ['node-exporter:9100']
Then reload Prometheus with docker-compose exec prometheus kill -HUP 1 and see what happens here: http://localhost:9090/targets.
PromQL documentation:
-
basic: https://prometheus.io/docs/prometheus/latest/querying/basics/
-
advanced: https://prometheus.io/docs/prometheus/latest/querying/functions/
Go to http://localhost:9090/graph and write a query displaying a graph of free memory on your OS.
Metric name is node_memory_MemFree_bytes.
💡 Solution
Query: node_memory_MemTotal_bytes{}
Same metric but in GigaBytes ?
💡 Solution
Query: node_memory_MemTotal_bytes{} / 1024 / 1024 / 1024
Same metric, but in percent of total available memory ?
Tips: node-exporter metrics are prefixed by node_.
💡 Solution
Query: (node_memory_MemTotal_bytes{} - node_memory_MemFree_bytes{}) / node_memory_MemTotal_bytes{} * 100
Uncomment grafana in docker-compose.yml and launch it:
docker-compose up -d grafana
Open http://localhost:3000 (user: admin / pass: workshop).
Add a new Prometheus datasource to Grafana.
- Name : Prometheus
- Pointing to http://prometheus:9090
Advance
You can use provisioning for create dashboards, datasources,..
Add a new dashboard to Grafana.
Create a graph showing current memory usage.
💡 Solution
Query: (node_memory_MemTotal_bytes{} - node_memory_MemFree_bytes{}) / node_memory_MemTotal_bytes{} * 100
Grafana should be displaying graph in %, such as:
💡 Solution
In the same dashboard, add a new graph for CPU load (1min, 5min, 15min).
Tips: you will need a new metric prefixed by node_.
💡 Solution
In the same dashboard, add a new graph for sda disk usage (ko written per second).
You will need rate() PromQL function: https://prometheus.io/docs/prometheus/latest/querying/functions/#rate
💡 Solution
Query:
rate(node_disk_written_bytes_total{device="sda"}[30s])
Let's import a dashboard from Grafana website.
- "Node Exporter Full" dashboard: https://grafana.com/dashboards/1860
- Or "Node Exporter Server Metrics" dashboard: https://grafana.com/dashboards/405
- Or both ;)
Those dashboards are only compatible with Prometheus data-source and node-exporter.
Uncomment postgres, postgresql-exporter and nginx-exporter services in docker-compose.yml, and launch containers.
docker-compose up -d nginx-exporter
docker-compose up -d postgres postgresql-exporter
Update Prometheus configuration to scrape Nginx and PostgreSQL exporters.
💡 Solution
scrape_config:
[...]
- job_name: 'postgresql-exporter'
static_configs:
- targets: ['postgresql-exporter:9187']
- job_name: 'nginx-exporter'
static_configs:
- targets: ['nginx-exporter:9101']Then docker-compose exec prometheus kill -HUP 1
Check everything is working well here: http://localhost:9090/targets
Take a look on /metrics routes of exporters: http://localhost:9187/metrics + http://localhost:9101/metrics
Send tens of requests to Nginx on localhost:8080 (200, 404...) and fill PostgreSQL database:
# 2xx
./infinite-200-req.sh
# 4xx
./infinite-404-req.sh# inserts data into pg
./infinite-pg-insert.shGo on https://grafana.com/dashboards and find a dashboard for PostgreSQL, compatible with Prometheus and wrouesnel/postgres_exporter.
💡 Solution
Those exporters looks nice: https://grafana.com/dashboards/6742, https://grafana.com/dashboards/6995.
Display 2 graphs:
-
number of 2xx http requests per second
-
number of 4xx http requests per second
Tips: you should use sum by(<label>) (<metric>) and irate(<metric>) (cf PromQL doc).
💡 Solution
Query graph 1: sum by (status) (irate(nginx_http_requests_total{status=~"2.."}[1m]))
Legend graph 1: Status: {{ status }}
Query graph 2: sum by (status) (irate(nginx_http_requests_total{status=~"4.."}[1m]))
Legend graph 2: Status: {{ status }}
Let's display in real time:
- number of users
- number of posts per user
Grab custom metrics with postgresql-exporter by adding queries to custom-queries.yml:
- Metric
user_countof typecounter=>SELECT COUNT(*) FROM users; - Metric
post_per_user_countof typegaugewith user_id and email in labels =>SELECT u.id, u.email, COUNT(*) FROM posts p JOIN users u ON u.id = p.user_id GROUP BY u.id;
Example and syntax here.
http://localhost:9187/metrics should output:
[...]
# HELP user_count_count Number of users
# TYPE user_count_count counter
user_count_count 2
# HELP post_per_user_count_count Number of posts per user
# TYPE post_per_user_count_count gauge
post_per_user_count_count{email="foobar@gmail.com",id="e1c10ca1-60c8-405c-a9f3-3ff41456ca9f"} 1
post_per_user_count_count{email="samuel@grep.to",id="fde08ee6-5fb9-4c4f-9b40-dc2ad69bb855"} 2
[...]
💡 Solution
Append to custom-queries.yml:
user:
query: "SELECT COUNT(*) FROM users;"
metrics:
- count:
usage: "COUNTER"
description: "Number of users"
post_per_user:
query: "SELECT u.id, u.email, COUNT(*) FROM posts p JOIN users u ON u.id = p.user_id GROUP BY u.id;"
metrics:
- count:
usage: "GAUGE"
description: "Number of posts per user"
- id:
usage: "LABEL"
description: "User id"
- email:
usage: "LABEL"
description: "User email"
docker-compose restart postgresql-exporter
With user_count{} and post_per_user_count{id,email} metrics, build following graphs:
Simple graph of users signup (rate(<metric>)):
Heatmap of signups (increase(<metric>)):
docker-compose exec grafana grafana-cli plugins install petrslavotinek-carpetplot-panel
docker-compose restart grafana
Table of top 10 users per post count (topk(), sum by(<label>) (<metric>)):
💡 Solution
Query : rate(user_count{}[1m])
Query : increase(user_count{}[$__interval]) > 0
Use Carpet Plot plugin
Display Aggregate : sum
Query: topk(10, sum by (id, email) (post_per_user_count{}) > 0)
Legend format : User {{ id }} - {{ email }}
Format as: time series. Option Table transform :Time series aggregation
You can play with this sample in Python or Springboot: instrumenting-code.md.
Uncomment example-springboot and example-python in docker compose and run docker-compose up --build.
Don't forget to update Prometheus configuration in prometheus.yml !
- Monitor a Redis server, a RabbitMQ cluster, Mysql...
- Increase data retention (15d by default).
- Setup alerting with AlertManager and basic rules
- Setup Prometheus service discovery (consul, etc, dns...) to import configuration automatically