Reference · ingest

Ingest reference

Everything an ingest point will accept, what it refuses, and exactly how it behaves when you send more than it will take. Four protocols, one authentication scheme, one set of limits.

All endpoints are TLS 1.3 only, HTTP/2 preferred, on the hostname of your account’s region. Replace the region label in every example with your own — the full list is on the regions page.

Endpoints

Paths are identical at every region; only the hostname differs.
Protocol	Host	Port	Path	Encodings
OTLP / gRPC	ingest.halkon.sbs	4317	—	gzip, zstd, identity
OTLP / HTTP protobuf	ingest.halkon.sbs	4318	/v1/logs, /v1/metrics, /v1/traces	gzip, zstd
OTLP / HTTP JSON	ingest.halkon.sbs	4318	/v1/logs, /v1/metrics, /v1/traces	gzip, zstd
Prometheus remote write v1	ingest.halkon.sbs	443	/api/v1/prom/write	snappy (required)
syslog RFC 5424 / RFC 5425	ingest.halkon.sbs	6514	—	identity (octet framing)
HTTP batch JSON / NDJSON	ingest.halkon.sbs	443	/v1/ingest/logs, /v1/ingest/metrics	gzip, zstd, identity
Native agent stream	ingest.halkon.sbs	443	/v2/stream	zstd (required)

Scroll the table sideways on a narrow screen. Hostnames resolve to the ingest point of that region only; there is no global anycast name, by design — see why there is no global endpoint.

Authentication

One header, on every protocol that has headers. Ingest keys carry the prefix hk_ik_, are scoped to a region and a signal set, and never grant read access.

HTTP headerall HTTP and gRPC transports

Authorization: Bearer hk_ik_REPLACE_WITH_YOUR_INGEST_KEY
Content-Type:  application/x-protobuf
Content-Encoding: zstd
X-Lp-Source:   "web-07.prod.internal"   # optional, overrides host detection

For syslog, which has no header envelope, the key is presented as the TLS client identity or inside the [halkon@0 key="..."] structured-data element on the first message of the connection. The second form keeps the key in your rsyslog template rather than in a certificate; the first is preferred for anything long-lived.

Key rotation

Two keys may be active per scope at once so rotation never needs a maintenance window. Deactivating a key closes its open streams within 30 s; agents reconnect with whatever key is still valid in their config.

Hard limits

These are enforced at the ingest point, not advisory. Anything over a limit is rejected with a specific code rather than silently truncated.

Per-account overrides above these values are available on the Regional plan.
Limit	Value	Applies to	On violation
request_body_compressed	8 MiB	all HTTP transports	413
request_body_decompressed	64 MiB	all HTTP transports	413
events_per_batch	10 000	OTLP, HTTP batch	422, batch rejected
samples_per_write	50 000	remote write	422
single_event_bytes	1 MiB	all	422, event dropped
log_body_bytes	512 KiB	logs	truncated, flagged
syslog_message_bytes	64 KiB	syslog	truncated, flagged
attributes_per_event	128	all	422
attribute_key_bytes	128	all	422
attribute_value_bytes	8 KiB	all	truncated, flagged
active_series_per_account	5 000 000	metrics	429 on new series only
timestamp_past_window	72 h	all	422
timestamp_future_skew	5 min	all	422
request_timeout	30 s	all HTTP transports	connection closed
concurrent_streams_per_key	256	HTTP/2, gRPC	REFUSED_STREAM
connection_idle_timeout	300 s	all	GOAWAY

“Flagged” means the event is stored with lp.truncated=true and the original byte length in lp.original_bytes, so a query can find what was cut.

Batching

A batch is the unit of acknowledgement. It either lands whole or it does not land at all; there is no partial acceptance inside a batch, which is what makes at-least-once delivery from the agent tractable.

The cost of a batch is dominated by its fixed overhead until roughly 1 000 events, and by compression after about 6 000. Batches between 2 000 and 5 000 events sit in the flat part of the curve; that is where the agent aims by default.

Recommended batch shape

events_per_batch2 000 – 5 000

compressed_size256 KiB – 2 MiB

flush_interval2 s (or on size)

in_flight_batches4 (per stream)

codeczstd level 3

Do not send one event per request

A single-event POST costs the same connection work as a 5 000-event one and will hit the per-key rate limit long before it hits the per-account ingest ceiling. If your producer cannot batch, put the agent in front of it.

Fig. 5 — Measured on the eu-north-1 ingest fleet with mixed JSON logs at 412 B mean. Your own curve will shift with event size but keeps this shape.

Response codes

Every rejection is specific enough to act on without opening a support ticket. The body is always JSON, even for protobuf requests.

Code	Meaning	Retry	What to do
202	Batch accepted and durably written	—	Drop the batch from your buffer. Receipt id in `lp-receipt`.
400	Malformed protobuf, JSON or framing	no	Fix the producer. Retrying will not help and counts against the key rate limit.
401	Missing or unparseable key	no	Check the `Authorization` header format.
403	Key valid, wrong region or wrong signal	no	Key scope mismatch. Body names the scope that was required.
413	Body over the size limit	no	Split the batch. Body reports both compressed and decompressed size.
422	Semantically invalid events in an otherwise valid batch	no	Body lists up to 20 offending event indices and the rule each broke.
429	Rate or series ceiling reached	yes	Honour `Retry-After`. Do not tighten the loop.
499	Client closed before acknowledgement	yes	The batch may or may not have landed. Resend; dedup will handle it.
500	Unclassified ingest fault	yes	Back off. If it persists past two minutes, check status.
503	Ingest point shedding load	yes	Back off. `Retry-After` is always present and always honoured by the agent.
507	Account storage quota exhausted	conditional	Data is being refused, not queued. Raise the quota or shorten retention.

Rejection body

HTTP 422application/json

{
  "accepted": 0,
  "rejected": 3,
  "receipt": "r-8f31c2a0",
  "region": "eu-north-1",
  "errors": [
    { "index": 41,  "rule": "attributes_per_event",  "got": 174, "limit": 128 },
    { "index": 902, "rule": "timestamp_past_window", "got": "2026-09-13T04:02:11Z" },
    { "index": 903, "rule": "single_event_bytes",     "got": 2361104, "limit": 1048576 }
  ]
}

Generated at — by the example renderer on this page — the timestamp above inside the payload is a fixed illustration of a stale event.

Retries and backoff

Retryable codes are 429, 499, 500, 502, 503, 504 and any transport-level failure. Everything else is a client bug and must not be retried.

base_delay1.0 s

multiplier1.8

max_delay60 s

jitter± 20 % (full range, uniform)

max_attempts12 then buffer to disk

retry_after_winstrue

Twelve attempts from a 1 s base with a 1.8 multiplier and a 60 s ceiling spans about 6 minutes 20 seconds of wall clock. After that the batch moves to the disk buffer and the agent keeps collecting; it does not block the producer and it does not discard.

Jitter is not optional. Without it, 58 200 agents that lost a shared ingest point come back in lockstep and re-create the outage. The agent’s reconnect jitter is deliberately wide for the same reason.

Nominal delay before each attempt, jitter not applied.
Attempt	Delay	Elapsed	State
1	1.0 s	1.0 s	retrying
2	1.8 s	2.8 s	retrying
3	3.2 s	6.0 s	retrying
4	5.8 s	11.8 s	retrying
5	10.5 s	22.3 s	degraded
6	18.9 s	41.2 s	degraded
7	34.0 s	75.2 s	degraded
8	60.0 s	135.2 s	degraded
9	60.0 s	195.2 s	degraded
10	60.0 s	255.2 s	degraded
11	60.0 s	315.2 s	degraded
12	60.0 s	375.2 s	buffering

Buffer behaviour, buffer sizing and drain order: agent → disk buffer.

Duplicates

Delivery is at-least-once, so duplicates are expected after any retry across an ambiguous failure. The ingest point deduplicates on a batch fingerprint for a bounded window; anything outside that window is stored twice and is your problem to filter at query time.

dedup_keysha-256(key_id, source, first_ts, last_ts, event_count, body_hash)

dedup_window15 min

dedup_scopeper ingest point (not cross-region)

duplicate_response202 with lp-duplicate: true

billing_of_duplicatesnot charged (within the window)

A deduplicated batch still returns 202 so a naive client cannot get stuck. The lp-duplicate header is informational and the agent uses it only for its own counters.

Backpressure and shedding

When an ingest point approaches its accept ceiling it sheds in a fixed order, and it tells you which class it is shedding through the lp-shed-class response header. Nothing is shed silently.

Order	Class	Typical content	Shed at
1	debug	DEBUG and TRACE severity logs, unsampled spans	88 % of ceiling
2	bulk	Explicitly marked bulk imports and backfill	92 % of ceiling
3	info	INFO severity logs	96 % of ceiling
4	metrics	Metric samples, oldest scrape first	98 % of ceiling
5	critical	WARN, ERROR, FATAL and anything marked `lp.keep=true`	never

Shed events return 503 with Retry-After, so a compliant client buffers rather than loses. An agent in buffering state is the intended outcome of shedding, not a failure.

Burst allowance

Every account may exceed its ingest ceiling by 3× for up to 10 minutes in any rolling hour without shedding and without an overage charge. A launch spike is normal; a permanently raised floor is a plan change.

Minimal examples

HTTP batch, NDJSON

shellone request, two events

# Each line is one event. Body is zstd-compressed NDJSON.
printf '%s\n' \
  '{"ts":"2026-09-01T09:14:02.881Z","severity":"ERROR","service":"checkout-api","body":"upstream timeout","attrs":{"pod":"checkout-api-7d9","status":504}}' \
  '{"ts":"2026-09-01T09:14:02.913Z","severity":"INFO","service":"checkout-api","body":"retry scheduled","attrs":{"pod":"checkout-api-7d9","attempt":1}}' \
  | zstd -3 \
  | curl --http2 -sS -X POST \
      -H 'Authorization: Bearer hk_ik_REPLACE_ME' \
      -H 'Content-Type: application/x-ndjson' \
      -H 'Content-Encoding: zstd' \
      --data-binary @- \
      https://ingest.halkon.sbs/v1/ingest/logs

Prometheus remote write

prometheus.ymlfragment

remote_write:
  - url: https://ingest.halkon.sbs/api/v1/prom/write
    authorization:
      type: Bearer
      credentials_file: /etc/prometheus/halkon.key
    queue_config:
      capacity: 10000
      max_samples_per_send: 5000
      batch_send_deadline: 2s
      min_backoff: 1s
      max_backoff: 60s
    write_relabel_configs:
      # keep the invoice honest: drop per-container churn
      - source_labels: [__name__]
        regex: 'container_(network|fs)_.*'
        action: drop

syslog over TLS

rsyslog.confRFC 5424 with octet framing

module(load="omfwd")
global(
  DefaultNetstreamDriver="gtls"
  DefaultNetstreamDriverCAFile="/etc/ssl/certs/ca-certificates.crt"
)
action(
  type="omfwd"
  target="ingest.halkon.sbs"
  port="6514"
  protocol="tcp"
  StreamDriver="gtls"
  StreamDriverMode="1"
  StreamDriverAuthMode="x509/name"
  StreamDriverPermittedPeers="ingest.halkon.sbs"
  Template="RSYSLOG_SyslogProtocol23Format"
  TCP_Framing="octet-counted"
  # queue so a lost link does not block the logging pipeline
  queue.type="LinkedList"
  queue.filename="halkon-fwd"
  queue.maxDiskSpace="512m"
  queue.saveOnShutdown="on"
  action.resumeRetryCount="-1"
)

More configurations, including an OpenTelemetry Collector exporter block and the agent’s own file: documentation → configuration examples.