Incidents

Rendered from the repository's runbook. Commands that use the Caddy admin API are in the Caddy guide; from Go, the same operations are Engine.SetMode and Engine.Purge.

SetMode always takes a time limit, at most 24 hours, so a mode nobody remembers to undo ends on its own.

4.1 The origin is down or slow

The breaker counts origin-health failures over a 10 s window (500 and 4xx are not failures unless Breaker.CountStatus500), opens at a 50% failure ratio of at least 20 outcomes, and stays open 5 s (±20%), doubling to 60 s on each consecutive reopen. Meanwhile entries inside their stale-if-error window are served with detail=stale-if-error or circuit-open; everything else gets 503.

If entries have no stale-if-error window and you want to ride out the outage: Engine.SetMode(weir.ModeStaleOnError, ttl) serves any stored entry stale (up to 24 h past expiry, but see the retention note below), except hard-purged, invalidated and must-revalidate, proxy-revalidate, no-cache or s-maxage ones. ttl is required, at most 24 h, and the mode reverts by itself. Entries are kept only for their stale windows plus Freshness.Keep (5 min) after expiry, and only when they carry ETag or Last-Modified, so the practical reach is minutes unless you raised Freshness.Keep or DefaultStaleIfError before the incident. Modes are not persisted across restarts. Set it from an admin handler, not from request data.

4.2 The cache itself is suspect

Wrong content is being served, or you changed the key configuration. SetMode(weir.ModeBypass, ttl) makes every request a pass-through (still through the limiter and breaker, nothing stored, existing entries untouched; an unsafe request that succeeds still invalidates its target as usual). Then purge (4.3) and leave bypass when it is clean. Check weir_not_stored_total and the origin's Vary and Cache-Control first: most wrong-content reports are an origin that varies on something Weir does not key (R-1, R-5, R-6).

4.3 Purging

Engine.Purge(ctx, weir.Purge{...}) takes All, URLs (absolute, rewritten like requests) and Groups (with Origin), in PurgeSoft (default) or PurgeHard mode.

• Soft: matching entries become stale at the purge time (with the memory store it can land up to 1 s late) and are revalidated with conditional requests. Inside the entry's stale-while-revalidate window clients keep getting the stale copy while a background refresh runs; with no such window (the default) the next request revalidates in the foreground and waits. Either way a purge of thousands of keys is bounded by coalescing per key and the limiter, and background refreshes never use the foreground reserve. Use it for content changes.
• Hard: matching entries are unusable at once, and clients wait for the origin. Use it for content that must not be served again (a leak, a takedown).

A same-origin 2xx or 3xx response to an unsafe method (POST, PUT, DELETE) also invalidates its target URI and Location and Content-Location URIs. Cache-Group-Invalidation from the origin soft-purges groups unless CacheGroups.Ignore is set. Purge errors reach you; a store that refuses a hard purge at its cap is not an outage.

4.4 Miss-rate anomaly

The log line weir: miss-rate anomaly and weir_miss_rate_anomalies_total mean that one partition (origin plus path) had at least 500 misses in a 10 s window at a miss ratio of 0.9 or more. Causes, in order of how often they happen:

1. A busy path whose responses are uncacheable (private, no-store) or always revalidated (max-age=0, no-cache). Every request is a miss by design. Route that path around Weir in your server (mount it on the mux beside the Weir handler); Bypass handles cookies and headers, not paths. It is not an attack and the warning will not go away until you do.
2. A cold path: a fresh deploy, a purge, a crawler walking many distinct URLs.
3. Cache busting with unique query strings (T-11). The per-path cap (Limiter.MaxPerPartition, 16) already bounds the damage; the flood cannot fill the shared queue.

MissRate.Throttle is off by default. Turn it on only after running without it and finding the warnings are cases 2 or 3. With it, an anomalous partition is capped at one origin fetch for the next window, for as long as it stays over both thresholds. Read the cost before you do (06 R-8):

• A cold path with far more distinct keys than a single fetch at a time can fill in a window stays capped. Measured: 300 000 distinct keys at 1 000 requests a second were capped in all 30 windows of the run.
• A client that sends max(500, 9 × the path's hits) distinct-query requests per window keeps that path capped for as long as it sends. In a review experiment, 60 requests a second let 390 of 1 200 legitimate misses through, against all 1 200 without Throttle (not a repo test).
• Sheds from a full global limiter count as misses, so a slow origin can get a busy, miss-heavy path capped in the next window.
• An attacker who floods every other window is never throttled.

Throttle trades the path's own misses for origin protection. If case 1 applies, fix it with a route, not with Throttle. MissRate.MinMisses and MinRatio tune detection, and TopK (64) bounds memory; a partition with under 1/TopK of a window's requests may go untracked.

4.5 Memory

weir_store_bytes should sit near the configured size once the cache is warm. If the process nears GOMEMLIMIT, look at Storable.MaxObjectBytes (1 MiB, headers included) and at large streamed bodies before you blame the cache: buffering for storage is bounded by MaxConcurrent × MaxObjectBytes (NFR-4, 64 MiB at defaults). A stream that exceeds MaxObjectBytes is not stored, but it holds up to MaxObjectBytes + 1 bytes of read-ahead after its fetch slot is released, until the client reads it. Many slow clients on large responses therefore cost about one MaxObjectBytes each (1 MiB at defaults), beyond the bound above. The hit path's GC cost at 1M entries is in benchmarks.md.

4.6 A remote store fails

After 5 consecutive ErrUnavailable results the store breaker opens for 1 s, doubling to 30 s. While it is open, lookups are treated as misses and nothing is stored, so origin load rises to the uncached rate. The limiter and breaker still protect the origin. Epoch (purge) state may be lost with a store restart; run a hard purge of everything (All: true) after one if stale content matters (06 R-4).

Errors clients see