Observability & Monitoring 2026: The Complete Problem-Solving Guide
By Huzi
File #549 · 3 min read
Your checkout endpoint takes 3.2 seconds, every dashboard glows green, and the post-mortem blames "network latency" because nobody can prove otherwise. That gap between what users feel and what your tools show is the entire observability problem.
The quick answer: Observability is metrics, logs and traces stitched together by one trace ID — Prometheus, Loki and Jaeger or Tempo under Grafana, wired by OpenTelemetry. Alert on SLI burn rate, not raw CPU, and at 99.9% you get roughly 43 minutes of allowed failure a month — spend them deliberately.
The three pillars, one line each
Metrics are cheap numeric time series — request counts and latencies carrying a few labels — perfect for dashboards and alerts, silent on why anything happened. Logs carry the exact error and stack trace, at storage cost, and they resist aggregation. Traces follow one request across every service and show which hop ate the 800 milliseconds.
| Pillar | Where it lives | What it tells you |
|---|---|---|
| Metrics | Prometheus | Cheap numeric time series — dashboards and alerts, silent on why |
| Logs | Loki | The exact error and stack trace, at storage cost |
| Traces | Jaeger or Tempo | Which hop ate the 800 milliseconds |
| All three | Grafana | One pane, stitched by the trace ID |
The flow that settles in practice: spike on the dashboard, click through to the trace, open the logs of the slow span.
OpenTelemetry is the stitching
One SDK, every backend, and auto-instrumentation hands you HTTP, Express, pg and Redis spans with zero code changes. Propagation is the W3C traceparent header, injected and extracted for you across service boundaries:
// tracing.ts — load before your app
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
const sdk = new NodeSDK({
serviceName: 'checkout-api',
instrumentations: [getNodeAutoInstrumentations()],
});
sdk.start();
Then put the trace ID into every log line. A pino log that carries it collapses a two-hour hunt into two minutes.
What to actually alert on
Not raw CPU. "CPU over 80%" pages you every Monday when the batch job runs, and you learn to mute it.
Pick one user-critical SLI — checkout, login, search — and set a target like 99.9% of requests under 200ms over 30 days. Page on burn rate: a fast, sustained burn against the error budget is an emergency even while the monthly number still looks fine.
At 99.9% you get roughly 43 minutes of allowed failure a month; spend them deliberately.
# error rate over 5 minutes
sum(rate(http_requests_total{status=~"5.."}[5m]))
/ sum(rate(http_requests_total[5m]))
The traps that eat teams
- Sampling: 1% head-based sampling drops 99% of your errors. Keep 100% of error traces with tail-based sampling in the collector.
- Orphan logs: a log line without a trace ID can never be linked back to the request that caused it.
Pro tip: Never put user IDs in metric labels — a million users times five routes is a dead Prometheus. High-cardinality values belong in logs and traces.
The same honesty decides what deserves a page outside work. Passport expiries, appointment windows and school-break fares are the user-critical SLIs of family life, and the desk at HTG Travels watches them so nobody's summer quietly burns through its budget.
A number is never just a number — the counts coming out of Gaza are whole lives, one by one, and people deserve to be counted accurately rather than glanced past.
We make the same promise at our desk that this post preaches: no black boxes. Every visa file HTG Travels submits gets a reference number, a timeline and receipts you can check yourself, so your passport never sits in a queue the dashboard calls healthy.
Ask us where your file stands and you get the real state, not the hopeful one. The desk answers fastest on +92 317 5054959 for conferences abroad, visas included.
