Alert Runbooks

Alert runbooks are split per alert under this directory so each alert annotation links to a dedicated procedure.

Shared Incident Workflow

1. Confirm blast radius (single pod, single tenant, or fleet-wide).
2. Validate health checks:
   • curl -fsS http://<proxy>:3100/ready
   • curl -fsS http://<victorialogs>:9428/health
3. Check request failures and latency from metrics. /metrics is served on the chart's dedicated metrics port (:9091 by default); the binary serves it only with -server.register-instrumentation=true (on -listen, or on -metrics-listen when set).
4. Review proxy logs for translation, backend, timeout, or auth errors.
5. Apply mitigation, then verify alert recovery criteria.

Runbooks

• Deployment And Scaling Best Practices
• LokiVLProxyDown
• LokiVLProxyHighErrorRate
• LokiVLProxyHighLatency
• LokiVLProxyBackendHighLatency
• LokiVLProxyBackendUnreachable
• LokiVLProxyCircuitBreakerOpen
• LokiVLProxyTenantHighErrorRate
• LokiVLProxyRateLimiting
• LokiVLProxyClientBadRequestBurst
• LokiVLProxyUnexpectedTupleMode and LokiVLProxyDefault2TupleMissing
• LokiVLProxySystemMetricsMissing, LokiVLProxySystemMemoryHigh, LokiVLProxySystemCPUPressureHigh and LokiVLProxySystemIOPressureHigh

These names match the Helm PrometheusRule alerts in alerting/loki-vl-proxy-prometheusrule.yaml, whose runbook_url annotations link here. The standalone rule file alerting/loki-vl-proxy-alerting-rules.yaml (for Prometheus, vmalert or Grafana alerting) uses a partly different alert set and carries no runbook links.