Reference · alerting
Alerting
A rule is an HKQL query, a threshold, a duration and a route. It is evaluated on a fixed schedule against the same storage your dashboards read, and it fires a webhook or an email. There is no separate alerting data path and no separate agent.
Rule format
Rules are YAML. They live in your account, can be managed through the API, and are versioned — every edit keeps the previous body so you can see what the rule looked like when it fired last Tuesday.
id: checkout-5xx-rate description: "checkout-api returning 5xx above 2% of requests" severity: critical # info | warning | critical interval: 30s for: 3m # condition must hold this long before firing resolve_after: 5m # and stay false this long before resolving query: | from logs | where ts > ago(5m) and service == "checkout-api" | summarize total = count(), bad = countif(status >= 500) by pod | extend ratio = bad * 1.0 / total | where total > 50 condition: ratio > 0.02 group_by: [pod] # one alert instance per pod annotations: summary: "{{ pod }} at {{ ratio | percent }} 5xx" runbook: "https://wiki.internal/runbooks/checkout-5xx" route: - channel: webhook-oncall when: severity == "critical" - channel: email-platform when: always throttle: 15m silence: - cron: "0 2 * * SUN" duration: 2h reason: "weekly maintenance window"
| Field | Type | Default | Meaning |
|---|---|---|---|
| id | string | — | Stable identifier. Appears in every notification and in the history. |
| severity | enum | warning | Routing input only. It does not change evaluation. |
| interval | duration | 15s | How often the query runs. Minimum 15 s, maximum 24 h. |
| for | duration | 0 | Condition must be continuously true this long before firing. |
| resolve_after | duration | = for | Condition must be continuously false this long before resolving. |
| query | HKQL | — | Must be cost class A or B and must complete within the interval. |
| condition | expression | — | Evaluated per result row. A row that matches is an alert instance. |
| group_by | list | [] | Columns that identify an instance. Empty means one instance per rule. |
| annotations | map | {} | Templated strings carried into the notification payload verbatim. |
| route | list | — | Ordered. Every matching entry fires; there is no first-match-wins. |
| silence | list | [] | Cron windows during which the rule evaluates but does not notify. |
| enabled | bool | true | A disabled rule keeps its history and its version chain. |
countif, used in the example, is the conditional form of count — see the function table.
Evaluation timeline
The for duration exists to stop a single bad scrape from waking someone up. This is exactly what it does to a condition that flaps.
| State | Meaning | Notifies | Counted in history |
|---|---|---|---|
| ok | Condition false, no pending timer | no | no |
| pending | Condition true, for timer running | no | yes |
| firing | Condition held for the full for duration | yes | yes |
| recovering | Condition false, resolve_after timer running | no | yes |
| stale | Query returned no rows for the instance’s group for 3 intervals | configurable | yes |
| error | Query failed, timed out, or exceeded its cost class | yes | yes |
stale is the state that catches a stopped producer. If you want “tell me when the data stops”, set on_stale: notify rather than writing a rule that counts zero rows — a query that returns no rows cannot evaluate a condition on them.
Delivery
Two channel types, both of which you control end to end. We do not integrate with third-party incident tools directly; we send a well-specified POST and let your own router decide what to do with it.
| Channel | Transport | Retries |
|---|---|---|
| webhook | POST, JSON, HMAC signed | 3 |
| SMTP from your region | 3 | |
| syslog | RFC 5424 over TLS to your collector | 3 |
After three failures the notification stops being retried and is written to the rule’s history with the last response code and body. It is also the trigger for the built-in notification_delivery_failed rule that every account gets by default.
Webhook payload
{
"version": 3,
"event": "firing",
"rule_id": "checkout-5xx-rate",
"rule_version": 11,
"severity": "critical",
"region": "eu-north-1",
"instance": { "pod": "checkout-api-7d9f4" },
"fingerprint": "a41e0c7b9d2f6183",
"started_at": "2026-09-01T09:17:30Z",
"evaluated_at": "2026-09-01T09:20:30Z",
"values": {
"total": 4188,
"bad": 312,
"ratio": 0.0745
},
"annotations": {
"summary": "checkout-api-7d9f4 at 7.45% 5xx",
"runbook": "https://wiki.internal/runbooks/checkout-5xx"
},
"query_link": "https://query.halkon.sbs/q/a41e0c7b9d2f6183"
}
lp-signature: sha256=<hex> lp-timestamp: 1788599430 lp-rule-id: checkout-5xx-rate lp-attempt: 1 lp-delivery: d-6f2ac91e
Verify with HMAC-SHA256(secret, lp-timestamp + "." + body) and reject anything whose timestamp is more than five minutes from your own clock. Right now that cutoff would be — minus five minutes.
Keeping the volume down
Every mechanism here reduces notifications, never evaluations. The rule keeps running and the history keeps recording, so a silenced alert is still visible on the rule page.
One instance per group key
A rule with group_by: [pod] across 60 pods produces up to 60 independent instances, each with its own for timer and its own fingerprint. Removing group_by collapses them into one.
Repeat suppression per route
A route with throttle: 15m sends at most one notification per fingerprint per 15 minutes while the instance stays firing. The resolve notification is never throttled.
Cron windows and ad-hoc mutes
Recurring windows live in the rule. One-off silences are created through the API or console with a mandatory reason and a mandatory expiry — there is no permanent silence.
Suppress the downstream noise
A rule may name another rule as a dependency. While the parent is firing, the child evaluates and records but does not notify, which is how one bad database stops producing forty alerts.
Absence as a signal
Set on_stale: notify to be told when a group that was reporting stops reporting. This is the correct way to detect a dead producer.
Hard ceilings
1 000 instances per rule, 20 000 firing instances per account, 600 notifications per minute per channel. Exceeding any of them fires the built-in alerting_overload rule instead of flooding you.
Alerting on the pipeline itself
Every account gets four built-in rules at creation. They are ordinary rules: editable, disableable, visible in the same list as your own. They exist because the most common monitoring failure is monitoring that stopped and nobody noticed.
| Built-in rule | Interval | for | Fires when |
|---|---|---|---|
| agent_buffering | 60s | 5m | Any agent reports hkagent_state >= 2, that is buffering or shedding. |
| agent_silent | 60s | 10m | A source that reported in the last 24 h has sent nothing for 10 minutes. |
| ingest_rejected | 60s | 5m | Non-retryable rejections exceed 0.5% of accepted batches. |
| notification_delivery_failed | 60s | 0 | Any notification is parked after three failed attempts. |
The agent_buffering rule, in full
id: agent_buffering description: "an agent cannot reach its ingest point and is writing to disk" severity: warning interval: 60s for: 5m query: | from agent_health | where ts > ago(5m) | summarize state = last(hkagent_state), buffer_mb = last(buffer_bytes) / 1048576, capacity = last(buffer_max_bytes) / 1048576 by source | extend fill = buffer_mb / capacity condition: state >= 2 group_by: [source] on_stale: notify # an agent that stops reporting is itself the alert annotations: summary: "{{ source }} buffering, {{ buffer_mb }} MiB written ({{ fill | percent }} of buffer)" route: - channel: email-platform when: always throttle: 30m
If the ingest point is unreachable, the agent cannot report that it is buffering. That is what agent_silent and on_stale are for: the absence of the heartbeat is evaluated centrally, from data that is already stored, and does not depend on the agent being able to talk to us. The status page is the other half of that answer.