From nothing to a first query

1 · Quick start

1. Create an ingest key

   In the console, or through the management API with an existing admin key. A key is bound to one region and cannot be moved.

   shellmanagement API

   curl -sS -X POST \
     -H 'Authorization: Bearer hk_ak_REPLACE_ME' \
     -H 'Content-Type: application/json' \
     --data '{"name":"prod-hosts","scope":{"signals":["logs","metrics"],"source_prefix":"web-"}}' \
     https://api.halkon.sbs/v1/keys
   
   # response, shown exactly once:
   # {"id":"k-7f21","key":"hk_ik_....","region":"eu-north-1","created":"2026-09-08T11:02:44Z"}

2. Install the agent

   A static binary plus a unit file. Verify the checksum published with the release before installing; the installer does that for you and refuses on mismatch.

   shellDebian or RPM host

   # 1. put the key somewhere only the agent can read
   sudo install -d -m 0750 -o hkagent -g hkagent /etc/hkagent
   printf '%s' "$LP_INGEST_KEY" | sudo tee /etc/hkagent/ingest.key >/dev/null
   sudo chmod 0400 /etc/hkagent/ingest.key
   sudo chown hkagent:hkagent /etc/hkagent/ingest.key
   
   # 2. minimal config
   sudo tee /etc/hkagent/agent.yaml >/dev/null <<'YAML'
   agent:
     key_file: /etc/hkagent/ingest.key
     source: "${HOSTNAME}"
   egress:
     endpoint: ingest.halkon.sbs
   collect:
     host_metrics:
       enabled: true
       interval: 15s
     files:
       - path: /var/log/app/*.log
         format: json
   YAML
   
   # 3. start
   sudo systemctl enable --now hkagent
   systemctl status hkagent --no-pager

3. Or skip the agent entirely

   If you already run an OpenTelemetry Collector, add an exporter and you are done. Nothing else on the host changes.

   otel-collector.yamlexporter fragment

   exporters:
     otlphttp/halkon:
       endpoint: https://ingest.halkon.sbs:4318
       compression: zstd
       headers:
         Authorization: "Bearer ${env:LP_INGEST_KEY}"
       sending_queue:
         enabled: true
         num_consumers: 4
         queue_size: 5000
         storage: file_storage/halkon   # survive a restart
       retry_on_failure:
         enabled: true
         initial_interval: 1s
         max_interval: 60s
         max_elapsed_time: 0s              # never give up, let the queue hold
   
   processors:
     batch:
       send_batch_size: 4000
       send_batch_max_size: 8000
       timeout: 2s
   
   service:
     pipelines:
       logs:
         receivers: [otlp, filelog]
         processors: [batch]
         exporters: [otlphttp/halkon]
       metrics:
         receivers: [otlp, hostmetrics]
         processors: [batch]
         exporters: [otlphttp/halkon]

4. Confirm data is being accepted

   Three independent checks, in increasing order of confidence.

   shelllocal, then remote

   # a) what the agent thinks
   curl -s http://localhost:9411/metrics | grep -E '^hkagent_(state|batches_accepted_total|buffer_bytes)'
   
   # b) a hand-made event, acknowledged or refused on the spot
   printf '{"ts":"%s","severity":"INFO","service":"smoke","body":"hello from %s"}\n' \
     "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$(hostname)" \
     | curl -sS -i -X POST \
         -H "Authorization: Bearer $LP_INGEST_KEY" \
         -H 'Content-Type: application/x-ndjson' \
         --data-binary @- \
         https://ingest.halkon.sbs/v1/ingest/logs
   
   # c) read it back
   curl -sS -H "Authorization: Bearer $LP_QUERY_KEY" \
     -H 'Content-Type: application/json' \
     --data '{"query":"from logs | where ts > ago(5m) and service == \"smoke\" | limit 10"}' \
     https://query.halkon.sbs/v1/query

   Step (b) uses your host’s own clock. Right now that would be —, which must fall inside the −72 h / +5 min acceptance window.

5. Run a real query

   Once host metrics are flowing there is something worth asking about.

   HKQLpaste into the console

   from metrics
   | where ts > ago(1h) and name == "system_memory_available_bytes"
   | summarize free_gb = min(value) / 1e9 by source, t = bin(ts, 5m)
   | sort by free_gb asc
   | limit 20
   | render line

2 · Keys and scopes

Three key types, three prefixes, no overlap in capability. A key is shown once at creation and stored only as a salted hash; if it is lost, rotate rather than recover.

An admin key deliberately cannot read your telemetry. Separating “who can change the account” from “who can read the logs” is the point of having three types.

Scope fields

Rotation without a window

1. Create the new key with the same scope. Two keys per scope may be active at once.
2. Roll it out to hosts. The agent reloads key_file on SIGHUP without dropping its stream.
3. Watch last_used_at on the old key until it stops moving.
4. Revoke the old key. Open streams using it close within 30 s; anything still holding it reconnects and gets 401, which is exactly the signal you want.

3 · Configuration examples

Kubernetes DaemonSet

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: hkagent
  namespace: observability
spec:
  selector:
    matchLabels: { app: hkagent }
  template:
    metadata:
      labels: { app: hkagent }
    spec:
      serviceAccountName: hkagent
      containers:
        - name: hkagent
          image: hkagent:4.2.1
          args: ["--config=/etc/hkagent/agent.yaml"]
          resources:
            requests: { cpu: 50m,  memory: 64Mi }
            limits:   { cpu: 500m, memory: 192Mi }
          env:
            - name: NODE_NAME
              valueFrom: { fieldRef: { fieldPath: spec.nodeName } }
          volumeMounts:
            - { name: config,  mountPath: /etc/hkagent,        readOnly: true }
            - { name: key,     mountPath: /etc/hkagent/secret, readOnly: true }
            - { name: buffer,  mountPath: /var/lib/hkagent }
            - { name: varlog,  mountPath: /var/log,            readOnly: true }
      volumes:
        - { name: config, configMap: { name: hkagent-config } }
        - { name: key,    secret:    { secretName: hkagent-ingest-key, defaultMode: 256 } }
        - { name: buffer, hostPath:  { path: /var/lib/hkagent, type: DirectoryOrCreate } }
        - { name: varlog, hostPath:  { path: /var/log } }

The buffer must be a hostPath or a persistent volume. On an emptyDir the buffer dies with the pod, which defeats the only mechanism that survives an outage.

Sending from a CI job

# Build metadata as a single structured event.
cat > /tmp/build-event.json <<JSON
{"ts":"$(date -u +%Y-%m-%dT%H:%M:%SZ)",
 "severity":"INFO",
 "service":"ci",
 "body":"build finished",
 "attrs":{"pipeline":"$CI_PIPELINE","status":"$JOB_STATUS",
          "duration_s":$SECONDS,"commit":"$GIT_SHA"}}
JSON

curl -sS --retry 5 --retry-connrefused --retry-delay 2 \
  -H "Authorization: Bearer $LP_INGEST_KEY" \
  -H 'Content-Type: application/x-ndjson' \
  --data-binary @/tmp/build-event.json \
  https://ingest.halkon.sbs/v1/ingest/logs

Retention policy

policies:
  - name: audit-trail
    match: 'service == "payments" and attrs.audit == true'
    retain: 360d

  - name: errors
    match: 'severity in ("ERROR", "FATAL")'
    retain: 90d

  - name: host-metrics
    match: 'signal == "metrics" and name startswith "system_"'
    retain: 400d

  - name: default
    match: 'true'
    retain: 30d       # the included window, no extra charge

Lengthening a policy applies from the moment you save it and does not resurrect data already deleted. Shortening it takes effect at the next daily sweep, within 24 hours. Cost arithmetic: pricing → retention.

4 · Verifying ingest

Four counters answer almost every “is it working” question. They are exported by the agent on loopback and also shipped into your account, so you can read them either way.

5 · First query, annotated

from logs                                  # pick the source; exactly one per query
| where ts > ago(15m)                      # time first: this prunes whole blocks
| where service == "checkout-api"        # then an indexed attribute
| where body contains "timeout"          # raw-body filters go last, they are cost class C
| summarize n = count() by pod            # collapse to one row per pod
| sort by n desc                          # sorting is only legal after summarize or limit
| limit 20

Reorder those three where clauses and the query still returns the same rows, but it reads a different amount of data. HKQL does not reorder them for you — see what a query costs.

6 · Troubleshooting

7 · Glossary