Prometheus and Grafana Basics: Actually Seeing What Your Server Is Doing

Quick Answer

Prometheus collects and stores time-series metrics from your servers and apps by periodically pulling them over HTTP, and Grafana turns those metrics into dashboards and alerts you can actually look at. Together they answer 'is my server okay right now, and what did it look like an hour ago' in a way raw log files can't, because metrics are structured numbers you can graph and query instead of text you have to grep through.

Why "Just Check the Logs" Doesn't Scale

Logs are great for answering "what exactly happened at 3:14am when this one request failed" — they're detailed, they're specific to one event, and they're the right tool for debugging a particular incident after the fact. They're a much worse tool for answering "has CPU usage been creeping up all week" or "is our error rate right now higher than it normally is at this time on a Tuesday," because those questions need you to look at a trend over time across potentially millions of log lines, not one event.

Metrics are a different kind of data built specifically for that second class of question: numbers sampled at regular intervals and tagged with labels, stored as a time series. "CPU usage, sampled every 15 seconds, for the last 30 days" is a metric. Prometheus is the tool that collects those numbers and stores them efficiently; Grafana is the tool that turns them into a graph you can actually look at, or an alert that fires when a number crosses a threshold. Neither replaces logging — a mature setup has both, used for the questions each one actually answers well.

How Prometheus Actually Works: The Pull Model

Most monitoring intuition from outside this space assumes a push model — each service actively sends its metrics somewhere. Prometheus does the opposite: it pulls. You tell Prometheus a list of HTTP endpoints, called targets, to check periodically, and on a fixed interval, commonly every 15 seconds, Prometheus itself makes an HTTP request to each target's /metrics endpoint and reads back whatever plain-text metrics that service is currently exposing.

This has a genuinely useful side effect: if a target stops responding to Prometheus's scrape request, Prometheus knows immediately, because the pull just fails. A push-based system has to build separate "did this thing stop pushing" detection on top, since silence is ambiguous — it could mean the service died, or it could mean nothing changed. With pulling, Prometheus itself becomes the thing checking "is this alive," for free, as a side effect of collecting data.

Your own application code rarely talks to Prometheus directly. Instead, you either instrument your app with a Prometheus client library that exposes a /metrics endpoint itself, or, for things you don't control the code of, like a Linux host's CPU and memory or a Postgres database's internals, you run a separate small process called an exporter next to it — node_exporter for host-level OS metrics, postgres_exporter for database internals — whose entire job is translating that system's state into the plain-text format Prometheus expects on a /metrics endpoint.

Writing prometheus.yml

Prometheus itself is configured with a single YAML file. Here's a real configuration, checked for valid YAML syntax before inclusion, that scrapes Prometheus's own metrics, a host's metrics via node_exporter, and a Node.js application exposing its own metrics endpoint:

global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node_exporter"
    static_configs:
      - targets: ["node-exporter:9100"]

  - job_name: "my-node-app"
    metrics_path: /metrics
    static_configs:
      - targets: ["app:3000"]

global.scrape_interval is the default frequency Prometheus scrapes every target at, unless a specific job overrides it — 15 seconds is a common default that balances resolution against storage growth. Each entry under scrape_configs is one job: a named group of targets that get labeled with that job name in the resulting data, which is how you later filter a query down to "just metrics from my Node.js app" versus "just metrics from node_exporter." static_configs is the simplest way to list targets, a fixed list of host:port pairs, and it's what you'd use for a small, mostly-static setup. Larger, dynamic environments, with containers that come and go or autoscaling groups, typically use service discovery instead, where Prometheus asks something like Kubernetes or Consul for the current list of targets rather than you maintaining a static list by hand.

metrics_path defaults to /metrics if you don't specify it, which is why the first two jobs omit it — it's only spelled out on the third job for clarity, not because the value is different.

PromQL: Actually Querying Your Metrics

Every metric Prometheus collects is really a set of time series, each identified by a metric name plus a set of key-value labels, like node_cpu_seconds_total{cpu="0", mode="idle"}. PromQL, Prometheus's query language, is built around selecting and transforming these series. Typing a bare metric name gives you an instant vector — the most recent value of every series matching that name, at the current point in time:

node_cpu_seconds_total{mode="idle"}

That's rarely useful on its own for a counter metric, because node_cpu_seconds_total only ever increases — it's cumulative seconds spent in that mode since the machine booted, not a percentage. To get something meaningful like "CPU usage over the last few minutes," you need rate(), which takes a range vector, a metric with a time window in square brackets, and computes the per-second average rate of increase over that window:

rate(node_cpu_seconds_total{mode="idle"}[5m])

This gives you the average per-second rate of idle CPU time over the trailing 5 minutes, per CPU core, per machine. To turn that into an actual "percent of CPU that's busy" number people can read on a dashboard, you'd subtract the idle rate from 1 and aggregate across cores with avg by:

100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])))

avg by (instance) collapses the per-core series down to one number per machine, labeled by the instance label, averaging across cores first, and then the arithmetic converts an idle-time fraction into a busy-percentage. This three-step progression — bare metric, then rate() over a window, then an aggregation like sum by or avg by — covers a large fraction of the PromQL you'll actually write day to day for counter-type metrics, which is most of what exporters expose.

Connecting Grafana to Prometheus

Prometheus itself ships a basic built-in graphing UI, but it's meant for quick debugging queries, not for building dashboards you'd actually keep open on a wall monitor or share with a team. Grafana is the tool for that: it connects to Prometheus, among many other data sources, as a backend and gives you a proper dashboard-building UI, with panels, variables, and alerting layered on top.

You can add Prometheus as a data source by hand through Grafana's UI, under Connections then Data sources, but for anything beyond a personal experiment, it's worth provisioning it as code instead, so a fresh Grafana instance comes up already configured without a manual click-through. Here's a provisioning file, again syntax-checked before inclusion, that Grafana reads on startup from its provisioning directory:

apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    jsonData:
      timeInterval: 15s

access: proxy means Grafana's own backend makes the request to Prometheus's url on the dashboard viewer's behalf, rather than the viewer's browser calling Prometheus directly — this is the setting you want for almost every real deployment, since it means Prometheus never needs to be reachable from outside your network, only from Grafana. jsonData.timeInterval tells Grafana the underlying scrape interval, which it uses to avoid rendering a misleadingly smooth-looking graph from data that's actually only updated every 15 seconds.

Building Your First Dashboard Panel

A Grafana dashboard is a grid of independent panels, and each panel is, at its core, one or more PromQL queries plus a choice of visualization: time series graph, single stat number, gauge, table, and others. Creating a panel means picking the visualization type, typing in a query like the CPU-percentage one from earlier, and setting a legend format so the graph's lines are labeled meaningfully instead of showing raw label combinations.

A genuinely useful starting dashboard for almost any server or service tends to cover a small, consistent set of panels: CPU usage, using the rate()-based query above; memory usage, from node_memory_MemAvailable_bytes against node_memory_MemTotal_bytes if you're using node_exporter; request rate for an HTTP service, rate(http_requests_total[5m]) summed by status code to separate 2xx from 5xx; and request latency, from a histogram metric using histogram_quantile() to get, say, the 95th percentile. These four — CPU, memory, request rate, and latency — are close to what people sometimes call the golden signals of a service, alongside error rate, which the request-rate-by-status-code panel already captures, and they're a reasonable default to build first before you know exactly what's specific to your own system.

Alerting: Knowing Before Your Users Tell You

A dashboard only helps if someone is actively looking at it. Alerting is what closes that gap — a rule that evaluates a PromQL expression on a schedule and fires when the condition is true, sending a notification instead of waiting for a human to notice a graph trending the wrong way. Prometheus supports this natively through alerting rules:

groups:
  - name: node-alerts
    rules:
      - alert: HighCpuUsage
        expr: 100 * (1 - avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m]))) > 90
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "CPU usage above 90% on {{ $labels.instance }}"

The expr is the same kind of PromQL query you'd type into a dashboard panel; the for: 10m is what keeps a brief, harmless spike from paging someone at 2am — the condition has to stay true continuously for 10 minutes before the alert actually fires, not just be true for one single evaluation. Prometheus evaluates rules like this itself and hands firing alerts off to a separate component, Alertmanager, which handles routing, meaning who gets notified, grouping related alerts into one notification instead of ten, and silencing.

Grafana has its own, separate alerting system built into more recent versions, which can alert off any data source Grafana knows about, not just Prometheus, and manages notification routing inside Grafana itself rather than through a separate Alertmanager process. For a setup that's Prometheus-only, Prometheus's native alerting plus Alertmanager is the more standard path; Grafana-managed alerting becomes more attractive once you're mixing several different data sources and want one place to manage all the rules.

Frequently Asked Questions