OpenTelemetry in Production: Building an Observability Stack That Answers Questions
Collecting telemetry is easy. Building an observability stack that answers questions during an incident takes deliberate design.
Observability projects tend to produce impressive dashboards and disappointing incidents. Data volume grows, the bill grows faster, and during an outage engineers still fall back to tailing logs on a single service. The gap is not tooling; it is that telemetry was collected without deciding which questions it needed to answer.
Why OpenTelemetry became the default
Before OpenTelemetry, instrumentation was vendor-specific, so changing observability platforms meant re-instrumenting every service. OpenTelemetry decouples generation from analysis: applications emit standard traces, metrics, and logs, a collector processes and routes them, and the backend becomes a swappable decision. That optionality is worth real money at renewal time, and it removes the excuse for not instrumenting until a vendor is chosen.
Instrument for questions, not coverage
Start by writing down the questions you actually need answered under pressure:
- Which service or dependency caused this latency increase?
- Which customers or tenants are affected right now?
- Did this deploy cause the regression, and can we prove it?
- Where is the request spending its time end to end?
Each question implies specific instrumentation. "Which customers are affected" requires a tenant attribute propagated through the trace context. "Did this deploy cause it" requires version and deployment identifiers on every span and metric. Retrofitting these mid-incident is impossible, which is why the question list must come first.
The three signals and their jobs
| Signal | Job | Design guidance |
|---|---|---|
| Metrics | Detect and alert | Low cardinality, aggregated, cheap to retain long term |
| Traces | Localise the problem in a request path | High detail, sampled intelligently, short retention |
| Logs | Explain a specific occurrence | Structured, correlated by trace ID, aggressively filtered |
The correlation between them is what makes the stack usable. Every log line should carry the trace ID, and every trace should link to the metrics its service emits. Without correlation you have three disconnected tools and a slower incident.
Sampling and cardinality: the two cost levers
Cost problems in observability are almost always cardinality problems. A single attribute containing user IDs or full URLs can multiply a metric's series count by millions. Rules that hold up:
- Never put unbounded values — user IDs, session IDs, raw paths — into metric labels. Put them on spans instead.
- Normalise route templates so
/orders/12345becomes/orders/:id. - Use tail-based sampling: decide after the trace completes, keeping all errors, all traces above a latency threshold, and a small percentage of routine successes.
- Filter and aggregate at the collector, not the backend — you pay for what you ship.
A typical outcome of applying these is a 60 to 80 percent reduction in ingest with no loss of diagnostic capability, because the traces you dropped were the ones nobody would ever open.
Deploy the collector as infrastructure
Run the collector as a layer you control rather than exporting directly from applications. It gives you one place to enforce sampling policy, redact sensitive attributes before egress, add resource metadata such as environment and region, buffer during backend outages, and fan out to more than one destination during a vendor migration. Applications should know nothing about the backend.
Alert on symptoms, not causes
Alerting on CPU, memory, and queue depth produces noise and misses real outages. Alert on user-visible symptoms expressed as service level objectives: request success rate and latency at a defined percentile per critical journey. Then use error budgets to decide when to stop shipping features and fix reliability instead.
Keep the paging rule strict: if an alert does not require a human to act within minutes, it is a dashboard entry or a ticket, not a page. Alert fatigue destroys more reliability than any single misconfiguration.
Rollout sequence
Instrument the critical path first — the two or three services in every revenue-affecting request — using auto-instrumentation to get baseline traces quickly, then add manual spans around the operations that actually matter such as database calls, external API requests, and queue handoffs. Establish trace context propagation across service boundaries before expanding breadth; a broken propagation chain produces disconnected fragments that look like data but answer nothing. Only then extend to the long tail of internal services.
Frequently asked questions
What is OpenTelemetry?
A vendor-neutral standard and SDK set for generating, collecting, and exporting traces, metrics, and logs, separating instrumentation from the analysis backend so vendors can be changed without re-instrumenting.
How do we control cost?
Restrict cardinality, apply tail-based sampling that preserves errors and slow traces, filter and aggregate at the collector, and set retention per data class.
Do we need all three signals?
Yes, for different jobs: metrics detect and alert, traces localise, logs explain. Correlation between them is what makes them useful together.
Is auto-instrumentation enough?
It is the right starting point but not sufficient. Business-relevant attributes such as tenant, plan, and operation type must be added manually, and those are usually what you need during an incident.