# 0008. OpenTelemetry as the observability baseline

- **Status:** Accepted
- **Date:** 2026-05-26
- **Deciders:** Loma Group Platform
- **Related:** [`platform/observability/README.md`](../../platform/observability/README.md), [`docs/runbooks/slo.md`](../runbooks/slo.md), [`docs/runbooks/incident.md`](../runbooks/incident.md)

## Context

Multiple processes (.NET, NestJS, worker) need traces + metrics + logs with **correlation across services**. Each runtime has native logging libraries (Serilog, pino). Stitching them together by hand is unsustainable.

## Decision

- All apps export to an **OpenTelemetry Collector** via OTLP. The Collector fans out:
  - **Metrics** → Prometheus (scraped).
  - **Traces** → Tempo (or any OTLP-compatible backend).
  - **Logs** → Loki.
- **Correlation ID** (`X-Correlation-Id`) is the cross-system join key. Middleware generates one if missing; logger pushes it into Serilog `LogContext` / pino bindings; emitted on every `audit_logs` row and every BullMQ job.
- The OTel Collector **strips Authorization + Cookie headers** before forwarding (defense in depth).
- **SLOs** ([`docs/runbooks/slo.md`](../runbooks/slo.md)) reference Prometheus metrics directly — no separate metric pipeline.

## Alternatives considered

| Option                                          | Why rejected                                                        |
| ----------------------------------------------- | ------------------------------------------------------------------- |
| Vendor-only (Datadog / NewRelic)                | Lock-in; cost ramp; we want the option to self-host pieces.         |
| Native logging only (Serilog/pino) without OTel | No correlation across services without bespoke plumbing.            |
| Jaeger / Zipkin specifically                    | OTel can export to either; choosing one upstream locks the apps in. |

## Consequences

### Positive

- Apps emit OTel; the Collector decides where to send. Backend swap = config change, no code change.
- One correlation-id end to end (UI → API → worker → DB audit).
- Prometheus scrape compatibility means existing dashboards work.

### Negative

- More moving parts in local dev (the local-dev section in `platform/observability/README.md` runs them via docker run; not always running by default).
- Tempo/Loki retention is real disk; review monthly.

### Neutral

- Sentry stays for application-error tracking — it complements traces, doesn't replace them.

## Compliance

- Logs that flow into Loki must NOT contain Confidential or Restricted tier data (see `docs/compliance/data-classification.md`). The Collector header strip + app-side redaction enforce this.

## References

- OpenTelemetry: https://opentelemetry.io
- Serilog → OTLP: https://github.com/serilog/serilog-sinks-opentelemetry
