Observability Stack Explained: Prometheus, Grafana, and Loki Working Together
"We have observability" usually means someone installed Grafana and called it done. Metrics, dashboards, and logs each answer a different question, and understanding which is which is what makes the stack actually useful instead of just another tab you never open.
Prometheus answers "what is the current state, and how has it trended?"
Prometheus scrapes numeric time-series data — request counts, latency percentiles, memory usage — on a pull-based model, polling your services' /metrics endpoints on an interval. It's good at answering questions like "is p99 latency creeping up" or "did memory usage spike at 3am." It's bad at answering "what exactly happened at 3:02am" — that's not its job.
Grafana answers "how do I see it?"
Grafana doesn't store anything itself; it's a query and visualization layer over Prometheus (and Loki, and other data sources). The value isn't the pretty graphs — it's building dashboards that map to how your team actually thinks about the system, so an on-call engineer can go from "something's wrong" to "here's the service and the metric" in one glance instead of hunting through raw queries.
Loki answers "what exactly happened?"
Loki indexes logs the way Prometheus indexes metrics — cheaply, by label, not by full-text search of every line. The trick that makes it fit the same stack as Prometheus: you query Loki through the same Grafana interface, and you can jump from a metric spike straight to the logs from that service in that time window, without switching tools or mental models.
How they actually work together
A real incident flow looks like: Prometheus alert fires (error rate spiked) → Grafana dashboard shows which service and when → Loki query, scoped to that service and time range, shows the actual error messages. Each tool does the part it's good at; none of them tries to do the others' job.
Takeaway
The stack isn't three tools bolted together for coverage — it's a funnel. Metrics tell you something's wrong and roughly where. Dashboards get you there fast. Logs tell you exactly what happened. Skipping any one layer means falling back to guesswork for the part it would have covered.