Log streams: Splunk, Datadog, any HTTPS collector

Every assessment and every identification can go to your own log platform as it happens: Splunk, Datadog, or any HTTPS collector you run. There you can search, chart and alert on them beside the rest of your logs. A webhook sends one event per call, and /traffic is a summary you poll; neither of those is a log. A stream is.

Streams belong to one environment, like webhooks. They are created through the Management API with a key that holds webhooks:write, because a stream is another way of sending this environment's data to an address you chose — and, since a stream is a read sent elsewhere, the scope that reads each dataset it sends: decisions:read for assessments, identifications:read for identifications.

What a line holds

An assessment (dataset assessments):

{
  "requestId": "asr_…",
  "action": "challenge",
  "score": 60,
  "reasons": ["session:agentNotRun"],
  "method": "GET",
  "host": "shop.example",
  "path": "/cart",
  "ip": "203.0.113.9",
  "country": "GB",
  "asn": "AS5089",
  "userAgent": "Mozilla/5.0 …",
  "bot": { "result": "notDetected", "type": "notDetected" },
  "crawler": { "status": "verified", "bot": "googlebot" },
  "signedAgent": "chatgpt-agent"
}

An identification (dataset identifications):

{
  "requestId": "…",
  "visitorId": "…",
  "visitorFound": true,
  "confidence": 0.99,
  "suspectScore": 12,
  "decision": "allow",
  "host": "shop.example",
  "path": "/checkout",
  "ip": "203.0.113.9",
  "country": "GB",
  "browser": "Chrome",
  "os": "Windows",
  "bot": { "result": "notDetected", "type": null },
  "tampering": false,
  "vpn": false,
  "proxy": false,
  "tor": false,
  "incognito": false,
  "virtualMachine": false,
  "developerTools": false
}

Some things are never in a line:

• The query string. It carries tokens and email addresses as often as it carries anything.
• Account, card, phone number, code or message. An assessment only ever held these as digests.
• The address, when the organization pseudonymizes them. ip is null then.

Destinations

Splunk ("kind": "splunk") goes to the HTTP Event Collector's /services/collector/event, authorised by an HEC token. Each line is one event with source antibot and a sourcetype of antibot:assessment or antibot:identification. Give the collector's base URL (Splunk Cloud's is https://http-inputs-<stack>.splunkcloud.com). The path is added for you unless the URL already names it. A query on the URL, such as an indexer acknowledgement ?channel=<guid>, is kept.

Datadog ("kind": "datadog") goes to the Logs intake of your site, authorised by an API key. Each line's fields become attributes, alongside ddsource:antibot, service:antibot, a dataset: tag and a message summary. The request's fields go under Datadog's standard attributes, so its facets and pipelines read them: http.method, http.url_details.host and .path, http.useragent and network.client.ip. A top-level host would be read by Datadog as the machine that sent the log.

HTTPS ("kind": "https") sends newline-delimited JSON (application/x-ndjson). It is signed the way webhooks are: antibot-signature: t=<ms>,v1=<hex>, an HMAC-SHA256 over <t>.<body> with the stream's secret. Each batch also carries an antibot-batch-id, the same on the batch's second try, so a collector that took the first and answered too late can drop the repeat. The secret is returned once, when the stream is created.

The address rules are the webhooks' rules: https, on the default port, to a public address. A self-hosted Splunk listening on 8088 needs a public HTTPS address on 443 in front of it.

Create one

curl https://api.antibot.dev/api/management/v1/log-streams \
  -H "Authorization: Bearer $ANTIBOT_MANAGEMENT_KEY" \
  -H "Content-Type: application/json" \
  -d '{"kind": "splunk", "url": "https://http-inputs-acme.splunkcloud.com", "token": "…"}'

Field
kind	splunk, datadog or https.
url	Splunk and HTTPS: where to send. Datadog: not given.
token	Splunk: the HEC token.
site	Datadog: datadoghq.com (the default), us3.datadoghq.com, us5.datadoghq.com, datadoghq.eu, ap1.datadoghq.com, ap2.datadoghq.com or ddog-gov.com.
apiKey	Datadog: an API key.
datasets	["assessments", "identifications"] (the default), or either one.
enabled	true by default.
environment	Which environment's logs, as for every Management API write.

The token and the API key are kept to send with, and never shown again: a listing has no credential in it. To rotate one, PATCH the stream with the new token or apiKey. A change reaches every server instance within 10 seconds: until then, one may still send with what it held.

• GET /api/management/v1/log-streams lists the streams.
• GET /log-streams/{id}, PATCH /log-streams/{id} (enabled, datasets, token, apiKey) and DELETE /log-streams/{id} read, change and remove one.
• POST /log-streams/{id}/test sends one test line now, in the stream's own format, and answers with what the destination said: {"delivered": false, "error": "HTTP 403: …"}.

Delivery

Each server instance batches lines per environment. A batch goes out when it holds 500 lines or 5 seconds after its first line, whichever comes first. A batch the destination refuses is tried once more. If it fails again, it is dropped and counted.

A request that records a line sends its own environment's batches behind its answer, never another environment's: one organization's slow collector costs another nothing. A timer sends what a quiet environment has left. One send takes at most 2,000 lines (four batches); a backlog goes on with the next.

While a destination is down, an instance holds at most 10,000 lines per environment. Past that, the oldest are dropped and counted. A collector that is down therefore costs the API memory up to that cap, and nothing else.

Every stream keeps a count of the lines it delivered, those that failed and those dropped — including lines lost while this service's own database was out of reach, counted once it is back. It also keeps when it last delivered and the last error. The logs themselves are never stored here.

From the first send that delivers nothing and fails, a stream carries failingSince, until one delivers — or a test line (POST /api/management/v1/log-streams/{streamId}/test) is taken; a test line refused does not mark it. A quarter of an hour in, the organization's owners and admins are told by mail, once, and it is in the audit log as log_stream.failing with the last error. That is sooner than a webhook is told of: a webhook's failures are retried for most of a day, a stream's lines are dropped, so every minute it stays down is lines missing from the destination. A paused stream is not told of.

Lines held in an instance's memory are lost if the instance stops before they go out, as they are with a log shipper whose disk fails. A stream is the live feed. The record stays the Server API and, for refusals, their references.