Reference · agent

hkagent — the process that never stops talking

One static binary per host. It opens a connection to your region’s ingest point, keeps it open, and spends the rest of its life moving bytes up that stream. This page is about what it costs you to run and what it does when the link goes away.

current_release4.2.1 — 2026-09-08, —

binary_size24.1 MiB static, no runtime dependency

config_reloadSIGHUP without dropping the stream

privilegesunprivileged user, no CAP_* required

outbound_onlyTCP 443 — listens on nothing by default

Resource footprint

Measured on a 4-core host with the default configuration, sustained for 24 hours at each rate. The agent allocates a fixed pool at start and does not grow with throughput until the in-memory queue is under pressure.

Linux x86-64, glibc, agent 4.2.1, zstd level 3, 412 B mean event.
Events/s	RSS steady	RSS peak	CPU (1 core)	Uplink	Disk writes
100	18 MiB	21 MiB	0.04 %	6 KiB/s	0
1 000	23 MiB	29 MiB	0.31 %	57 KiB/s	0
5 000	31 MiB	38 MiB	1.42 %	280 KiB/s	0
20 000	46 MiB	58 MiB	5.10 %	1.1 MiB/s	0
50 000	74 MiB	96 MiB	12.8 %	2.7 MiB/s	occasional
100 000	118 MiB	128 MiB	26.4 %	5.4 MiB/s	continuous

“Disk writes” here means spill to the buffer while the link is healthy — it starts once the in-memory queue cannot drain fast enough, which on a 4-core host is around 60 000 events/s.

Memory ceiling

The hard cap is a configured number, not a hope. At memory_limit_mb the agent stops accepting into memory and spills everything to the disk buffer; if the buffer is also full it applies the drop policy you configured, and it always tells you which.

memory_limit_mb128 (default)

min_supported32

in_memory_queue60 % of the limit

compression_arena24 MiB fixed

oom_behaviourspill, never grow

Fig. 6 — During the outage the in-memory queue fills to its 60% share, then the agent spills and memory stops climbing. It never approaches the cap.

The disk buffer

A write-ahead log of fixed-size segments in one directory. Events are appended compressed, in arrival order, with a checkpoint written every 4 MiB. A crash loses at most the tail of the open segment.

When the link comes back the agent drains the buffer oldest first, interleaved with live traffic at a configurable ratio, so a long outage does not starve the present while it catches up on the past.

buffer_dir/var/lib/hkagent/buffer

buffer_max_mb512 (default)

buffer_range64 MiB – 64 GiB

segment_size32 MiB

checkpoint_every4 MiB

fsync_policyon segment close

drain_ratio3:1 live : backlog

overflow_policydrop_oldest | drop_lowest_severity | block

How long 512 MiB lasts

The buffer stores compressed bytes, so its capacity in events depends on what you send. These are the numbers we use when sizing a fleet.

Outage survivable on the default 512 MiB buffer.
Events/s	Mix	Buffered	Survives
100	host metrics only	0.9 KiB/s	6.6 d
500	metrics + light logs	9 KiB/s	15.8 h
2 000	typical app host	44 KiB/s	3.3 h
5 000	chatty JSON logs	128 KiB/s	68 min
20 000	debug enabled	560 KiB/s	15 min
100 000	aggregation node	2.9 MiB/s	2.9 min

Sizing rule

Pick the longest outage you want to survive without loss, multiply by your buffered byte rate, add 25%. For an aggregation node the honest answer is usually a larger buffer and a smaller drop_lowest_severity threshold rather than hoping.

What happens when the link goes down

Five states, deterministic transitions, every transition logged to the host’s own log and counted in the agent’s self-metrics. Nothing here is heuristic.

Fig. 7 — The only lossy state is SHEDDING, and reaching it takes a full buffer, which takes a real outage. Everything else is a delay, not a loss.

State	Entered when	Data path	Left when	Self-metric
connected	Stream established, acknowledgements arriving	memory queue → network	Any batch fails	hkagent_state=0
degraded	1 to 11 consecutive batch failures	memory queue grows, retries in flight	Acknowledgement, or attempt 12	hkagent_state=1
buffering	12 consecutive failures, or memory queue at 60% of the limit	memory queue → disk segments	A reconnect attempt is acknowledged	hkagent_state=2
draining	Link restored while the buffer is non-empty	disk + live, interleaved 1:3	Buffer empty	hkagent_state=3
shedding	Buffer at `buffer_max_mb`	`overflow_policy` decides what is discarded	Buffer falls below 90% full	hkagent_state=4

The state value is exported on the agent’s local metrics endpoint and also shipped as an event, so you can alert on hkagent_state >= 2 from inside Halkon — see alerting on agent health.

Reconnect behaviour

The connection is HTTP/2 over TLS 1.3, one stream per signal type, held open indefinitely. A keepalive ping every 20 seconds detects a silently dropped path faster than TCP will.

keepalive_ping20 s

ping_timeout10 s

reconnect_base1 s

reconnect_max60 s

reconnect_jitter± 40 %

stream_max_age4 h then graceful re-establish

dns_recheckevery 60 s while disconnected

clock_skew_guardrefuse to send if host clock moves > 5 min

Why the jitter is so wide

Reconnect jitter is ±40%, far wider than the retry jitter, because a lost ingest point means tens of thousands of agents trying to come back at the same instant. A narrow window turns recovery into a second outage.

Proxies and egress control

Many fleets cannot open arbitrary outbound connections. The agent supports an explicit forward proxy and refuses to fall back to a direct path if the proxy is unreachable.

/etc/hkagent/agent.yamlegress section

egress:
  endpoint: ingest.halkon.sbs
  proxy: http://proxy.internal:3128
  proxy_required: true    # never bypass
  tls:
    min_version: "1.3"
    ca_file: /etc/ssl/certs/ca-certificates.crt
    server_name: ingest.halkon.sbs
  keepalive: 20s
  reconnect:
    base: 1s
    max: 60s
    jitter: 0.4

The endpoint is always a hostname. The agent resolves it at connect time and re-resolves every 60 seconds while disconnected, so an ingest point can be moved without touching a single host.

Full configuration

One file. Anything not present takes the default shown on this page. SIGHUP applies changes to collection, filtering and buffering without dropping the stream; changes under egress re-establish it.

/etc/hkagent/agent.yamlcomplete example

agent:
  key_file: /etc/hkagent/ingest.key      # mode 0400, owned by hkagent
  source: "${HOSTNAME}"
  tags:
    env: prod
    team: platform
    rack: "b14"

egress:
  endpoint: ingest.halkon.sbs
  keepalive: 20s

resources:
  memory_limit_mb: 128
  max_cpu_percent: 30           # soft; throttles compression, never drops

buffer:
  dir: /var/lib/hkagent/buffer
  max_mb: 512
  segment_mb: 32
  overflow_policy: drop_lowest_severity
  drain_ratio: 3

batch:
  max_events: 4000
  max_bytes: 1MiB
  flush_interval: 2s
  compression: zstd
  compression_level: 3

collect:
  host_metrics:
    enabled: true
    interval: 15s
    sets: [cpu, memory, disk, filesystem, network, load]
  files:
    - path: /var/log/app/*.log
      format: json
      multiline: false
      severity_field: level
    - path: /var/log/nginx/access.log
      format: regex
      pattern: '^(?P<remote>\S+) \S+ \S+ \[(?P<ts>[^\]]+)\] "(?P<req>[^"]*)" (?P<status>\d{3})'
  journald:
    enabled: true
    units: ["ssh.service", "nginx.service"]
  othk_receiver:
    enabled: false        # set true to accept OTLP from local processes on the loopback interface

filters:
  - drop: 'severity == "DEBUG" and env == "prod"'
  - redact:
      field: body
      pattern: '(?i)(authorization|cookie|set-cookie):\s*\S+'
      replacement: '\1: [redacted]'
  - sample:
      match: 'severity == "INFO" and service == "healthcheck"'
      rate: 0.01

self:
  metrics_listen: localhost:9411   # loopback only
  log_level: info
  ship_self_metrics: true

Redaction happens before the wire

The redact filter runs on the host, inside the agent, before compression and before anything leaves the machine. If a pattern matches, the original never exists in our storage — there is nothing to purge later.

Platforms and packaging

Platform	Architectures	Packaging	Service manager	Support
Linux, glibc ≥ 2.17	x86-64, arm64	deb, rpm, tar.gz	systemd	tier 1
Linux, musl	x86-64, arm64	apk, tar.gz	OpenRC, s6	tier 1
Container image	x86-64, arm64	OCI, distroless	DaemonSet	tier 1
Windows Server 2019+	x86-64	msi, zip	Windows service	tier 2
FreeBSD 13+	x86-64	pkg, tar.gz	rc.d	tier 2
macOS 13+	arm64	tar.gz	launchd	dev only

Tier 1 means release-blocking tests on every commit. Tier 2 means tested at release. “Dev only” means we build it and use it ourselves but do not carry a support obligation.

Release history

Version	Date	Age	Change
4.2.1	2026-09-08	—	Drain ratio made configurable; journald cursor fix on unit restart.
4.2.0	2026-08-12	—	Reconnect jitter widened to ±40%; self-metrics now shipped by default.
4.1.4	2026-06-24	—	zstd arena no longer grows under sustained spill.
4.1.0	2026-04-02	—	Local OTLP receiver; `proxy_required` flag.
4.0.0	2026-01-20	—	Segment format v2. In-place upgrade from 3.x drains the old buffer first.

Agent releases are supported for 18 months. An agent more than 18 months old still ingests; it simply stops receiving fixes. Known defects are listed on the status page.