All six ingest points are accepting

Ingest backpressure at the platform peak

At 13:42 UTC the eu-north-1 ingest fleet crossed its shed threshold while the platform was at its all-time peak of 3.10 M events/s. Shedding proceeded in the documented order and reached the info class, which meant correctly formed INFO-severity batches were refused with 503 for 41 minutes.

Cause: a single account raised its own send rate roughly ninefold inside twenty minutes after enabling debug logging across a large fleet, and the regional ceiling had 30% headroom rather than the 40% our own policy requires. The burst allowance absorbed the first ten minutes; after that the shed logic did what it is designed to do.

Agents buffered as designed and drained within 18 minutes of recovery. Measured data loss: zero for accounts with a non-full disk buffer; four accounts with buffers under 128 MiB and high event rates lost between 40 seconds and 6 minutes of INFO-severity logs.

Changed since: eu-north-1 ceiling raised from 900 000 to 1 200 000 ev/s on 2026-08-26. Per-account rate-of-change detection now warns the account and us when accepted volume rises more than 4× inside an hour. The capacity policy is now enforced as an alert rather than a review item.

Query latency after a block-index rebuild

A routine index rebuild on na-east-1 ran with the wrong concurrency setting and competed with interactive queries for disk bandwidth. Query p95 rose from 0.9 s to 11 s. Ingest was unaffected throughout and no batch was refused.

Alert rules whose queries exceeded their evaluation interval entered the error state and notified, which is correct behaviour and was the first signal we acted on.

Changed since: rebuild concurrency is now derived from measured idle bandwidth instead of a fixed constant, and rebuilds yield to interactive queries rather than sharing fairly with them.

Notification delivery backlog

An outbound path from sa-east-1 degraded and webhook deliveries began timing out at 10 s. Three delivery attempts per notification meant the queue grew faster than it drained; 1 184 notifications were parked, and every one of them was surfaced in its rule’s history and re-delivered manually after recovery.

Ingest, storage and query were unaffected. Alerts continued to evaluate and record correctly — only delivery was affected, which is the distinction the incident classification is built around.

Changed since: delivery workers are now per-channel rather than per-region, so one slow destination cannot occupy the pool. The built-in notification_delivery_failed rule was given a zero for duration so the first parked notification notifies immediately.

Query timeouts on wide scans

A change to the block pruner produced a plan that read every block in the time range instead of the ones matching the attribute predicate. Queries over more than roughly 200 GB hit the 60 s interactive timeout. Narrow queries, dashboards and all ingest were unaffected.

Changed since: the planner’s estimated block count is now compared against the actual count after every query, and a divergence above 20% fails the release gate.

Storage node replacement refused writes

During a planned node replacement the replacement node joined before its volume was fully attached. Batches requiring two-of-three durable copies could not be acknowledged and returned 503 for 18 minutes. Agents buffered; nothing was lost.

Changed since: a node is not admitted to the write quorum until it has served a synthetic read at full block size. The check adds about 90 seconds to every replacement.

Certificate rotation broke older syslog senders

A scheduled certificate rotation changed the issuing chain. Syslog clients validating against a pinned intermediate rather than the root failed TLS on port 6514 for 26 minutes until the previous chain was restored. OTLP, remote write and HTTP ingest were unaffected because those clients validate against the root.

Changed since: rotations now publish the new chain 30 days ahead and serve both for a 14-day overlap. The syslog listener logs, per sender, which chain was used to validate — visible in your own account as hk_ingest_tls_chain.