Reference · HKQL v3
HKQL
A pipeline of stages separated by |. Each stage takes a table and returns a table. There are no joins across regions, no subqueries in the where clause, and no way to write a query whose cost you cannot see before you run it.
from <source> | where <predicate> # narrow first, always | extend <name> = <expr> # derive columns | summarize <agg> by <keys> # collapse | sort by <column> desc | limit <n>
Sources are logs, metrics, traces and agent_health. A query without an explicit time predicate is rewritten to the last 15 minutes rather than scanning your retention window by accident.
Worked examples
1. Error rate per pod, one-minute buckets
from logs | where ts > ago(1h) and service == "checkout-api" | where severity in ("ERROR", "FATAL") | summarize errors = count() by pod, t = bin(ts, 1m) | sort by t asc, errors desc
| pod : string | t : timestamp | errors : int64 |
|---|---|---|
| checkout-api-7d9f4 | 2026-09-01T09:14:00Z | 318 |
| checkout-api-7d9f4 | 2026-09-01T09:15:00Z | 294 |
| checkout-api-2b81c | 2026-09-01T09:15:00Z | 11 |
| checkout-api-7d9f4 | 2026-09-01T09:16:00Z | 7 |
2. Latency percentiles from a histogram metric
from metrics | where ts > ago(6h) and name == "http_request_duration_seconds" | where route startswith "/api/" | summarize p50 = histogram_quantile(0.50, le, value), p95 = histogram_quantile(0.95, le, value), p99 = histogram_quantile(0.99, le, value) by route, t = bin(ts, 5m) | extend p99_ms = round(p99 * 1000, 1) | sort by p99_ms desc | limit 20
3. Pull a field out of unstructured text
from logs | where ts between ("2026-08-19T13:00:00Z", "2026-08-19T15:00:00Z") | where body contains "upstream timeout" # cheap filter first | extend upstream = regex_extract(body, 'upstream=([a-z0-9.\-]+)', 1) | where isnotnull(upstream) | summarize hits = count(), hosts = count_distinct(source) by upstream | sort by hits desc | limit 25
Putting regex_extract before the contains filter in the query above raises the scan cost by roughly 40× on the same data. HKQL executes stages in the order you write them, deliberately, so that the cost estimate you are shown is the cost you get.
4. Correlate a trace with the logs around it
from traces | where ts > ago(30m) and duration_ms > 2000 | summarize slow = count() by trace_id, service | lookup logs on trace_id within 30s | project trace_id, service, severity, body | limit 200
5. Agent fleet health
from agent_health | where ts > ago(5m) | summarize state = last(hkagent_state), buffer_mb = last(buffer_bytes) / 1048576 by source | where state >= 2 | sort by buffer_mb desc
Run against an account with 400 hosts this returns in well under a second: agent_health is a narrow, low-cardinality source held in a separate block family.
Pipeline operators
| Operator | Form | Effect | Notes |
|---|---|---|---|
| from | from <source> | Selects logs, metrics, traces or agent_health | Must be the first stage. Exactly one per query. |
| where | where <predicate> | Row filter | Pushed into the block index when it touches indexed attributes. |
| extend | extend <n> = <expr> | Adds computed columns | Evaluated per surviving row. Place after your cheap filters. |
| project | project <cols> | Keeps and orders columns | project-away removes instead. |
| summarize | summarize <aggs> by <keys> | Collapses rows into groups | Group count is capped at 100 000; exceeding it is an error, not a truncation. |
| sort by | sort by <col> asc|desc | Orders the result | Only after summarize or limit; sorting a raw scan is refused. |
| limit | limit <n> | Caps the row count | Default 1 000, maximum 50 000 for interactive, 1 000 000 for export. |
| lookup | lookup <src> on <key> within <d> | Time-bounded correlation | The only cross-source operator. within is mandatory and capped at 5 min. |
| mv-expand | mv-expand <col> | One row per array element | Rejected if it would multiply the row count beyond 20×. |
| render | render <kind> | Chart hint for the console | line, area, bars, table, heatmap. Ignored by the API. |
There is no join. Two sources meet only through lookup, which is bounded in time and therefore bounded in cost.
Function table
Cost class is what the planner charges before it runs anything: A is index-resident, B reads column data, C touches the raw body. Query concurrency and the per-query scan budget are plan limits, listed on the pricing page.
Aggregations
| Function | Signature | Sources | Cost | Notes |
|---|---|---|---|---|
| count | count() | all | A | Answered from block headers when no row filter applies. |
| count_distinct | count_distinct(col) | all | B | Exact below 216 values, then HyperLogLog with 1.6% error. |
| sum | sum(numeric) | all | B | Overflow raises an error rather than wrapping. |
| avg | avg(numeric) | all | B | Null values are skipped, not treated as zero. |
| min / max | min(col), max(col) | all | A | Block-header resident for timestamps and numerics. |
| percentile | percentile(col, q) | all | B | t-digest, relative error under 1% at the tails. |
| p50 / p90 / p95 / p99 | p99(col) | all | B | Shorthand for percentile at fixed q. |
| histogram_quantile | histogram_quantile(q, le, value) | metrics | B | Linear interpolation inside the bucket, as the convention expects. |
| stddev | stddev(numeric) | all | B | Population, not sample. Use stddev_samp for the other one. |
| rate | rate(counter, window) | metrics | B | Counter resets are detected and bridged. |
| delta | delta(gauge, window) | metrics | B | First minus last inside the window. No reset handling. |
| first / last | first(col), last(col) | all | B | By timestamp within the group, not by arrival order. |
| topk | topk(col, k) | all | B | Space-saving sketch; exact for k below 1 000. |
| any | any(col) | all | A | An arbitrary value from the group. Cheapest way to carry a column through. |
Scalar functions
| Function | Signature | Returns | Cost | Notes |
|---|---|---|---|---|
| now | now() | timestamp | A | Fixed once at plan time, so a long query stays self-consistent. |
| ago | ago(duration) | timestamp | A | ago(90m), ago(2d). Units: s, m, h, d, w. |
| bin | bin(ts, duration) | timestamp | A | Floors to the bucket. Buckets are aligned to the epoch, not to the query start. |
| bin_auto | bin_auto(ts) | timestamp | A | Picks a bucket that yields 60–200 points over the query range. |
| round | round(n, digits) | float64 | A | Half away from zero. |
| coalesce | coalesce(a, b, …) | any | A | First non-null argument. |
| iff | iff(pred, a, b) | any | A | Both branches must share a type. |
| isnull / isnotnull | isnull(col) | bool | A | Missing attribute and explicit null are the same thing here. |
| strlen | strlen(s) | int64 | B | UTF-8 code points, not bytes. bytelen gives bytes. |
| tolower / toupper | tolower(s) | string | B | ASCII-only fast path; falls back to full Unicode when needed. |
| substring | substring(s, start, len) | string | B | Zero-indexed. Out-of-range clamps instead of erroring. |
| split | split(s, sep) | array | B | Feed to mv-expand to get one row per element. |
| parse_json | parse_json(s) | dynamic | C | Access members with .. Malformed input yields null, not an error. |
| regex_extract | regex_extract(s, re, group) | string | C | RE2 syntax. No backreferences, no lookaround, guaranteed linear time. |
| regex_matches | regex_matches(s, re) | bool | C | Prefer contains or startswith when they will do. |
| parse_kv | parse_kv(s, pairsep, kvsep) | dynamic | C | For key=value key2=value2 log lines. |
| ipv4_in_subnet | ipv4_in_subnet(col, cidr) | bool | B | Operates on values you sent; we do not enrich or geolocate. |
| format_bytes | format_bytes(n) | string | A | Presentation only. Never use it before an aggregation. |
Operators == != < <= > >= in between contains startswith endswith matches and or not. String comparison is case-sensitive; append _cs-free variants such as contains_ci for case-insensitive matching.
What a query costs to run
Queries are not billed. They are, however, scheduled, and a query that would scan a quarter of your retention window is going to wait behind one that will not. The console shows the estimate before you press run.
The estimator is a block count, not a guess: block headers carry min and max timestamps plus a per-attribute value sketch, so the planner knows how many blocks a predicate can touch before reading any of them.
Running a query from a script
Query keys carry the prefix hk_qk_ and are read-only. The endpoint is the query host of your region, which is a different name from the ingest host and can be firewalled separately.
curl -sS --http2 \ -H 'Authorization: Bearer hk_qk_REPLACE_ME' \ -H 'Content-Type: application/json' \ -H 'Accept: application/x-ndjson' \ --data '{ "query": "from logs | where ts > ago(15m) and severity == \"ERROR\" | summarize n = count() by service | sort by n desc | limit 10", "format": "ndjson" }' \ https://query.halkon.sbs/v1/query
HTTP/2 200 lp-query-id: q-4c19ae72 lp-blocks-read: 1148 lp-bytes-scanned: 31742885888 lp-rows-emitted: 10 lp-elapsed-ms: 842 lp-cost-class: B lp-region: eu-north-1
Request issued from this page at — would carry lp-request-time equal to that value; the example above is a captured response and its ids are fixed.
Result formats
| format | Accept | Streaming | Use for |
|---|---|---|---|
| ndjson | application/x-ndjson | yes | Pipelines, anything that should not buffer the whole result. |
| json | application/json | no | Small results, dashboards, ad-hoc scripts. |
| csv | text/csv | yes | Spreadsheets. Header row carries the inferred types as a comment line. |
| arrow | application/vnd.apache.arrow.stream | yes | Exports above 100 000 rows and anything feeding a dataframe. |
| prom | text/plain; version=0.0.4 | no | Exposing a saved query to a scraper or a Grafana data source. |
Saved queries get a stable id and can be read by any tool that speaks one of these formats. That is the whole integration story — there is no plugin to install and nothing proprietary on the wire.