What you'll learn
Quick Answer
Logs record individual events, metrics record numbers over time, traces follow one request across services. Alert on symptoms users experience, not on causes, or you will drown in notifications and stop reading them.
Logs, metrics and traces
Three different tools, and using the wrong one is why people find monitoring frustrating.
Logs are individual events with detail. "User 42 failed to log in at 14:32:05 because the password did not match." Ideal for investigating a specific incident, and expensive to store and search at volume.
Metrics are numbers over time. Requests per second, error rate, response time, memory used. Cheap to store, easy to graph and alert on, and they tell you that something is wrong but never why.
Traces follow a single request through every service it touches, with timing for each hop. Essential once you have more than one service — see microservices vs monolith.
The usual workflow: a metric alerts you, a trace narrows down where, and logs tell you exactly what happened.
Logging that is actually useful
Use structured logs. A line of prose cannot be filtered; a JSON object can.
# hard to search
logger.info(f"User {user_id} failed login")
# searchable
logger.info("login_failed", extra={
"user_id": user_id,
"reason": "bad_password",
"ip": request.remote_addr,
"request_id": request_id,
})Now "show every failed login for user 42 today" is a query rather than a regular expression.
Include a request ID generated at the entry point and attached to every log line for that request. Without it, logs from concurrent requests interleave and you cannot tell which lines belong together. This single practice makes production debugging dramatically easier.
Use levels honestly. ERROR means something needs attention; WARN means something unusual; INFO means notable events; DEBUG is off in production. Logging everything at ERROR makes the level meaningless.
Never log secrets. Passwords, tokens, card numbers and personal data end up in log aggregation systems with far broader access than your database — a genuinely common and serious leak.
The metrics that matter
Rather than instrumenting everything, start with the four that describe user experience — commonly called the golden signals:
- Latency — how long requests take. Track percentiles, not averages.
- Traffic — requests per second.
- Errors — the rate of failed requests.
- Saturation — how full your resources are: CPU, memory, disk, connection pool.
Percentiles matter more than averages, and this is worth internalising. If 95% of requests take 100ms and 5% take 10 seconds, the average is around 600ms — a number describing nobody's experience. The p50 says 100ms, the p99 says 10 seconds, and the p99 is the users who are unhappy.
Averages hide exactly the problem you are looking for. Alert on p95 or p99.
Alerting people will actually read
Most alerting setups fail the same way: too many alerts, most not actionable, so people mute the channel. An ignored alert is worse than no alert, because it creates false confidence.
Two rules fix most of it.
Alert on symptoms, not causes. "Error rate above 5% for five minutes" is a symptom users feel. "CPU above 80%" is a cause that may be entirely fine — a busy server doing useful work. Alerting on causes produces constant noise; alerting on symptoms produces alerts that always matter.
Every alert must be actionable. If the response is "yes, we know, ignore it", delete the alert or fix the underlying issue. Alerts that are routinely ignored train people to ignore all of them, including the real one.
Use a duration condition so a brief blip does not page anyone, and separate urgency: things that need waking someone at 3am, and things that can wait for the morning. Very few things genuinely belong in the first category.
Where to start
For a small project, in order of value:
- Uptime monitoring. An external service checking your site every minute. Free, five minutes to set up, and it catches total outages — including ones your own monitoring cannot report because it is down too.
- Error tracking. A service that captures exceptions with stack traces and context, and groups duplicates. This is the highest-value tool for a small team by a wide margin.
- Structured logs with request IDs, shipped somewhere searchable.
- Basic metrics and a dashboard for the golden signals.
Two things worth adding early: a health endpoint that genuinely checks dependencies rather than returning 200 unconditionally, and a heartbeat check for scheduled jobs — failure alerts do not catch a job that stopped being scheduled at all. See background jobs and cron.
Monitoring only earns its cost when someone looks at it. Deciding who checks what, and when, matters as much as the tooling.