Grafana observability
These are the practices from a production-grade self-hosted stack: about 60 dashboards, 400+ alert rules, 50 scrape jobs, and logs and traces, all managed as code. Each rule below exists because breaking it caused a real incident. The why is given so you can judge edge cases.
Pick the mode first
- Learning ("teach me", "explain", "what is", "quiz me"): follow references/learning-path.md. Teach one stage at a time and run the lab in assets/lab/. Check understanding before moving on. Don't dump every stage at once.
- Building (a dashboard, alert, pipeline, Terraform or rollout): follow the workflow below and open only the reference file that the task needs.
- Planning a rollout at an organisation: references/implementation-plan.md.
The stack in one picture
apps / hosts / network kit / cloud APIs
│ metrics (exporters, /metrics) │ logs (files, docker, syslog) │ traces (OTLP)
▼ ▼ ▼
Prometheus ◄── scrape ── Grafana Alloy (collector: discover, relabel, redact, ship)
│ │ │
│ Loki Tempo
└──────────────┬────────────────────┴────────────────────────────────┘
▼
Grafana ── dashboards, Explore, alerting ──► contact points (chat, email, on-call)
▲
Terraform (folders, data sources, rules, routing) + scripts (dashboards) + git
Details: references/architecture.md.
The hard rules
- Code is the source of truth. Change alerting through Terraform, and build dashboards with a generator script or JSON that lives in git. A click in the UI is lost on the next apply, or worse, the next apply silently reverts it.
- Before any apply, check both kinds of drift. (a) Rules that exist live but not in code: an untargeted apply deletes them. (b) Rules that are in code but were changed live: an apply reverts them. A "No changes" plan proves nothing about rules that Terraform has never seen. See references/as-code.md.
- Guard the saved plan. Run
terraform plan -out, then refuse anydelete/replaceyou did not intend, on any resource. Putprevent_destroyon folders. Deleting a Grafana folder deletes every dashboard and rule in it. - Use a service-account token, never
admin:password. If the state was created under token auth, a basic-auth apply plans delete-and-recreate of folders, contact points and the policy tree. That once wiped 59 dashboards and the whole routing tree in a single apply. - Never put
notification_settings(a contact point) on a rule. A rule that has one skips the policy tree completely. It escapes severity routing, repeat intervals and mute timings. In one estate, 196 of 217 rules did this and nobody noticed. - Put the threshold in Grafana's threshold stage, not inside the PromQL. With
expr > 90, a healthy host returns no series, which Grafana reads as NoData, not OK. Query the raw value; compare it in stage C. - Route by actionability, and treat repeat intervals as the real lever. Use
severity(critical/warning/info) plusurgency=backlogfor chronic, known issues. Most alert noise is re-notification of something already known. Repeat critical alerts daily and warnings weekly, not hourly. - Escalation should be a delay, and a silence is the acknowledgement. The page route uses
group_waitof about 15m, so it pages only if the alert is still firing and unsilenced. Event-style rules needkeep_firing_for≥ the wait, or they resolve before they ever escalate. - Monitor the monitoring. Watch for evaluation failures, notification delivery failures, suppressed (silenced) alerts, stale exporters and backups that stopped. A silent alerting system looks exactly like a healthy one. See references/meta-monitoring.md.
- Verify, don't trust. Run every panel or alert query against the data source before writing it. After any API write, read the object back and assert that your change is in it: the POST response does not tell you what content went up. Agreement between two people or two sessions is not verification; one command is.
Workflow for any change
- State the question that the panel or alert answers, and who acts on it. If nobody would act on an alert, it's a dashboard panel, not an alert.
- Prove the query in Explore or through the data-source proxy API. Check that it returns data, that the labels are the ones you'll group or route by, and how it behaves when healthy (empty? zero?).
- Write it in code: a Terraform rule group, a dashboard generator, or an Alloy config. Follow references/alerting.md or references/dashboards.md.
- Check drift, then
terraform plan -out=tfplan. Read every collapsed~block, especially ongrafana_notification_policy, because it decides where every alert goes. - Apply the saved plan with a service-account token. Use
-targetwhen unsure. - Read it back from the API, confirm that the rule evaluates (state is not Error/NoData), and for alerts, fire a test end to end to the real contact point.
- Commit with the why. Leave a comment in the code wherever a choice looks wrong but is deliberate.
Definition of done
Alert rule: in Terraform; severity label set; summary and description templates include
the affected host or service and the current value; no_data_state chosen on purpose; for long
enough not to flap; routed by the tree (no per-rule contact point); a runbook link or a first step
in the description; tested end to end; it doesn't double-page with an existing rule.
Dashboard: in git (generated, or exported with id/version stripped); in the right folder;
linked from the overview hub and back again; variables for env/host; every panel has a title that
asks or answers a question; units and thresholds set; keepTime on links.
New exporter or data source: scraped with stable labels (env, team, service, role);
covered by the fleet up == 0 rule (don't add a separate one); if it has a last-run or freshness
metric, a staleness rule; added to the meta-monitoring audit; data source has a fixed uid.
Reference index
| File | Open it when |
|---|---|
| learning-path.md | teaching or learning, stage by stage with labs |
| architecture.md | choosing components, labels, retention, auth, HA |
| queries.md | writing PromQL or LogQL, or a query looks wrong |
| dashboards.md | building or reviewing a dashboard |
| alerting.md | rules, routing, contact points, silences, fatigue |
| as-code.md | Terraform, API, backups, drift, restores |
| meta-monitoring.md | proving the monitoring itself works |
| implementation-plan.md | introducing Grafana to a team or company |
Assets: assets/lab/ is a local Docker Compose lab. assets/terraform-starter/ is a starting Terraform stack with the safe defaults already wired in.