CommunityProductivity & Collaborationgithub.com

grafana-observability

Learn, design, build and run a Grafana observability stack (Prometheus metrics, Loki logs, Tempo traces, Alloy collection, Grafana dashboards and alerting) managed as code with Terraform. Use when asked to learn or teach Grafana, PromQL or LogQL; to set up monitoring; to write or review a dashboard, an alert rule, a notification policy, a contact point, an Alloy pipeline or Grafana Terraform; or to debug alerts that are noisy, silent, stuck in NoData or not reaching anyone.

What is grafana-observability?

grafana-observability is a Claude Code agent skill that learn, design, build and run a Grafana observability stack (Prometheus metrics, Loki logs, Tempo traces, Alloy collection, Grafana dashboards and alerting) managed as code with Terraform. Use when asked to learn or teach Grafana, PromQL or LogQL; to set up monitoring; to write or review a dashboard, an alert rule, a notification policy, a contact point, an Alloy pipeline or Grafana Terraform; or to debug alerts that are noisy, silent, stuck in NoData or not reaching anyone.

Works with~Claude Code~Codex CLI~Cursor
npx skills add https://github.com/louij2/grafana-observability-skill/tree/HEAD/.github/skills/grafana-observability

Installed? Explore more Productivity & Collaboration skills: steipete/gemini, steipete/gh-issues, steipete/skill-creator · View all 6 →

Ask in your favorite AI

Open a new chat with this agent skill pre-loaded.

Documentation

Grafana observability

These are the practices from a production-grade self-hosted stack: about 60 dashboards, 400+ alert rules, 50 scrape jobs, and logs and traces, all managed as code. Each rule below exists because breaking it caused a real incident. The why is given so you can judge edge cases.

Pick the mode first

  • Learning ("teach me", "explain", "what is", "quiz me"): follow references/learning-path.md. Teach one stage at a time and run the lab in assets/lab/. Check understanding before moving on. Don't dump every stage at once.
  • Building (a dashboard, alert, pipeline, Terraform or rollout): follow the workflow below and open only the reference file that the task needs.
  • Planning a rollout at an organisation: references/implementation-plan.md.

The stack in one picture

 apps / hosts / network kit / cloud APIs
      │ metrics (exporters, /metrics)     │ logs (files, docker, syslog)   │ traces (OTLP)
      ▼                                   ▼                                ▼
  Prometheus  ◄── scrape ──        Grafana Alloy (collector: discover, relabel, redact, ship)
      │                                   │                                │
      │                                 Loki                             Tempo
      └──────────────┬────────────────────┴────────────────────────────────┘
                     ▼
                 Grafana  ── dashboards, Explore, alerting ──►  contact points (chat, email, on-call)
                     ▲
                 Terraform (folders, data sources, rules, routing) + scripts (dashboards) + git

Details: references/architecture.md.

The hard rules

  1. Code is the source of truth. Change alerting through Terraform, and build dashboards with a generator script or JSON that lives in git. A click in the UI is lost on the next apply, or worse, the next apply silently reverts it.
  2. Before any apply, check both kinds of drift. (a) Rules that exist live but not in code: an untargeted apply deletes them. (b) Rules that are in code but were changed live: an apply reverts them. A "No changes" plan proves nothing about rules that Terraform has never seen. See references/as-code.md.
  3. Guard the saved plan. Run terraform plan -out, then refuse any delete/replace you did not intend, on any resource. Put prevent_destroy on folders. Deleting a Grafana folder deletes every dashboard and rule in it.
  4. Use a service-account token, never admin:password. If the state was created under token auth, a basic-auth apply plans delete-and-recreate of folders, contact points and the policy tree. That once wiped 59 dashboards and the whole routing tree in a single apply.
  5. Never put notification_settings (a contact point) on a rule. A rule that has one skips the policy tree completely. It escapes severity routing, repeat intervals and mute timings. In one estate, 196 of 217 rules did this and nobody noticed.
  6. Put the threshold in Grafana's threshold stage, not inside the PromQL. With expr > 90, a healthy host returns no series, which Grafana reads as NoData, not OK. Query the raw value; compare it in stage C.
  7. Route by actionability, and treat repeat intervals as the real lever. Use severity (critical/warning/info) plus urgency=backlog for chronic, known issues. Most alert noise is re-notification of something already known. Repeat critical alerts daily and warnings weekly, not hourly.
  8. Escalation should be a delay, and a silence is the acknowledgement. The page route uses group_wait of about 15m, so it pages only if the alert is still firing and unsilenced. Event-style rules need keep_firing_for ≥ the wait, or they resolve before they ever escalate.
  9. Monitor the monitoring. Watch for evaluation failures, notification delivery failures, suppressed (silenced) alerts, stale exporters and backups that stopped. A silent alerting system looks exactly like a healthy one. See references/meta-monitoring.md.
  10. Verify, don't trust. Run every panel or alert query against the data source before writing it. After any API write, read the object back and assert that your change is in it: the POST response does not tell you what content went up. Agreement between two people or two sessions is not verification; one command is.

Workflow for any change

  1. State the question that the panel or alert answers, and who acts on it. If nobody would act on an alert, it's a dashboard panel, not an alert.
  2. Prove the query in Explore or through the data-source proxy API. Check that it returns data, that the labels are the ones you'll group or route by, and how it behaves when healthy (empty? zero?).
  3. Write it in code: a Terraform rule group, a dashboard generator, or an Alloy config. Follow references/alerting.md or references/dashboards.md.
  4. Check drift, then terraform plan -out=tfplan. Read every collapsed ~ block, especially on grafana_notification_policy, because it decides where every alert goes.
  5. Apply the saved plan with a service-account token. Use -target when unsure.
  6. Read it back from the API, confirm that the rule evaluates (state is not Error/NoData), and for alerts, fire a test end to end to the real contact point.
  7. Commit with the why. Leave a comment in the code wherever a choice looks wrong but is deliberate.

Definition of done

Alert rule: in Terraform; severity label set; summary and description templates include the affected host or service and the current value; no_data_state chosen on purpose; for long enough not to flap; routed by the tree (no per-rule contact point); a runbook link or a first step in the description; tested end to end; it doesn't double-page with an existing rule.

Dashboard: in git (generated, or exported with id/version stripped); in the right folder; linked from the overview hub and back again; variables for env/host; every panel has a title that asks or answers a question; units and thresholds set; keepTime on links.

New exporter or data source: scraped with stable labels (env, team, service, role); covered by the fleet up == 0 rule (don't add a separate one); if it has a last-run or freshness metric, a staleness rule; added to the meta-monitoring audit; data source has a fixed uid.

Reference index

FileOpen it when
learning-path.mdteaching or learning, stage by stage with labs
architecture.mdchoosing components, labels, retention, auth, HA
queries.mdwriting PromQL or LogQL, or a query looks wrong
dashboards.mdbuilding or reviewing a dashboard
alerting.mdrules, routing, contact points, silences, fatigue
as-code.mdTerraform, API, backups, drift, restores
meta-monitoring.mdproving the monitoring itself works
implementation-plan.mdintroducing Grafana to a team or company

Assets: assets/lab/ is a local Docker Compose lab. assets/terraform-starter/ is a starting Terraform stack with the safe defaults already wired in.

Related Skills