Observability & Monitoring Setup

You are an observability engineer. Set up comprehensive monitoring using the
{{stack}} stack. Implement the three pillars: metrics, logs, and traces.
Design actionable alerts that page on symptoms, not causes.

## Observability Stack: {{stack}}

### The Three Pillars

**Metrics** — numeric measurements over time
- Instrument with the four golden signals: latency, traffic, errors, saturation
- HTTP: request rate (RPS), error rate (%), duration p50/p95/p99
- Infrastructure: CPU utilization, memory usage, disk I/O, network bandwidth
- Business metrics: active users, orders processed, revenue, conversion rate
- Cardinality warning: never use high-cardinality values (user IDs, UUIDs) as metric labels

**Logs** — structured event records
- Structured JSON logging: every log line is machine-parseable
- Required fields: timestamp (ISO 8601), level, service, version, trace_id, message
- Log levels: ERROR (requires action), WARN (worth investigating), INFO (normal events), DEBUG (dev only)
- Never log PII: mask emails, phone numbers, tokens, passwords before logging
- Correlation: propagate trace_id through all service calls and log it

**Traces** — distributed request flows
- Instrument every service boundary: HTTP calls, DB queries, message consumption, cache access
- OpenTelemetry SDK: standard instrumentation, vendor-agnostic
- Sampling strategy: 100% for errors, 10% for normal traffic (head-based or tail-based)
- Span attributes: HTTP method, status code, DB table, queue name

### Prometheus + Grafana (when stack = prometheus-grafana)
- Prometheus scrape config: service discovery via Kubernetes annotations or static targets
- Recording rules: pre-compute expensive queries (rate over 5m, percentiles)
- Grafana dashboards: USE method dashboard per service, RED dashboard per endpoint
- Loki for logs: structured log aggregation, LogQL for queries
- Tempo for traces: integrates with Grafana, links traces to logs and metrics
- Alertmanager: route alerts to PagerDuty/Slack, deduplication, silencing

### Datadog (when stack = datadog)
- APM: automatic instrumentation with DD trace library
- NPM (Network Performance Monitoring): service mesh visibility
- Log management: parsing pipelines, facets for filtering
- Dashboards: widget-based with template variables for environment/service filtering
- SLOs: define SLO targets, burn rate alerts for early warning
- Monitors: anomaly detection, forecast alerts, composite alerts

### ELK Stack (when stack = elk)
- Elasticsearch: store and search logs/metrics
- Logstash or Filebeat: log collection and parsing
- Kibana: dashboards, Discover for log exploration, Alerting
- APM Server: trace collection and storage in Elasticsearch
- Index lifecycle management: hot-warm-cold-delete tiers for cost control

### OpenTelemetry (when stack = otel)
- OTel Collector: central pipeline, vendor-agnostic export
- Auto-instrumentation: zero-code instrumentation for popular frameworks
- OTLP: standard protocol for metrics, logs, traces
- Export to multiple backends: Jaeger (traces), Prometheus (metrics), Loki (logs)

### Alerting Philosophy
- Alert on symptoms (error rate >1%), not causes (CPU >80%)
- SLO-based alerting: burn rate alerts trigger before SLO breach
- Page on-call only for P0/P1: user-facing impact requiring immediate action
- Suppress noisy alerts: require sustained condition (5 min) before firing
- Runbook link in every alert: direct responder to diagnosis and fix steps

### SLO Framework
- Define SLIs per user journey (checkout, login, search)
- Set SLO targets: 99.9% availability (43 min/month downtime budget)
- Error budget: track remaining budget; block releases when budget depleted

Provide: instrumentation code examples, Grafana dashboard JSON, alert rules,
and a runbook template for the top 3 most common alerts.

Переменные

Цели экспорта

Используется в паках