Observability
The Cache-Status header, Prometheus metrics, events, and which signals deserve an alert.
Weir reports every decision three ways: a header on the response, an event to an observer, and (with the observe/prom module) Prometheus metrics.
What to watch
Each response carries Cache-Status: Weir; ... (RFC 9211). hit; ttl=N is a fresh hit. fwd=uri-miss (no entry), fwd=vary-miss (entry exists, not for these Vary values), fwd=stale, fwd=request (the client asked), fwd=method and fwd=bypass say why the origin was asked. ; collapsed marks a request that joined another's fetch. detail=negative marks a negative-cache entry. A stale hit shows a negative ttl=. detail= names the stale reason: stale-while-revalidate, stale-if-error, shed, circuit-open, coalesce-timeout. The metric label reason uses short names for the first two: swr and sie.
With observe/prom (a separate module):
| Metric | Read it as |
|---|---|
weir_requests_total{outcome} | the hit ratio and the mix of outcomes |
weir_origin_fetches_total{class,result}, weir_origin_fetch_seconds | origin load and latency, split foreground, background and warm |
weir_origin_inflight, weir_limiter_queue_depth | the limiter: sustained inflight at MaxConcurrent with a growing queue means the origin is the limit |
weir_shed_total{reason} | requests refused (clients get 503 with Retry-After) |
weir_breaker_state (0 closed, 1 half-open, 2 open), weir_breaker_transitions_total{to} | origin health as Weir sees it |
weir_stale_served_total{reason} | how often staleness is hiding a problem |
weir_coalesce_timeouts_total{fallback}, weir_coalesced_total | flights that ran long, and how much coalescing saves |
weir_store_errors_total{op}, weir_store_bytes, weir_evictions_total{queue} | the store: errors mean it is unavailable, evictions are normal when full |
weir_not_stored_total{reason}, weir_key_rejections_total{reason} | responses refused by storability rules, and requests refused by validation |
weir_miss_rate_anomalies_total{action} | see 4.4 |
weir_purges_total{mode}, weir_negative_served_total | purge activity and negative-cache use |
Labelled series appear on their first event, so alert on absent() as well as rate(). Without Prometheus, implement weir.Observer (it runs on the request path: keep Observe fast) and poll Engine.Stats() for in-flight, queued, breaker state and store bytes. Request-derived strings such as Event.Partition are never metric labels.
On Caddy every weir_* series also carries a name label, the site's name, and the metrics are served by Caddy's own metrics handler.
Alerts
- The breaker is open for more than a few minutes (
weir_breaker_state == 2). weir_shed_totalis nonzero for more than a minute: the origin cannot keep up, orMaxConcurrentis set too low for it.weir_stale_served_total{reason="sie"}orreason="circuit-open"is rising: you are serving through an outage.weir_store_errors_totalis rising (remote stores only).- Hit ratio falls by half against the same hour last week.
As Prometheus rules, the first two look like this. Thresholds are starting points:
- alert: WeirBreakerOpen
expr: weir_breaker_state == 2
for: 3m
- alert: WeirShedding
expr: rate(weir_shed_total[1m]) > 0
for: 1mEvents without Prometheus
Config.Observer receives one weir.Event per decision. It is called synchronously on the request path, so it must be fast, must not block and must not panic. Engine.Stats() returns in-flight fetches, queue depth, breaker state and store bytes for gauges.
Event kinds may gain values in a minor release. Switch on them with a default case.
Limits
The
Cache-Statusheader never includes the cache key (T-27). Request-derived strings such as the partition are never metric labels, so a flood of distinct paths cannot explode your series count; the cost is that metrics do not tell you which path is misbehaving. The miss-rate warning in the log does.