Delivery model: confirms, retries, backpressure and the DLQ
Two stages on the message broker (RabbitMQ or Amazon SQS, see Message broker), both in the backend:
- Ingest and route.
POST /audit/publish(or/audit/publish/batch) redacts the event and publishes it to the ingest queue, then waits until the broker stored it (RabbitMQ publisher confirm, SQSSendMessage): 200 means the broker has it; nothing withinauditflow.broker.confirm-timeout(5 s) is a 503 the client retries with the sameeventId. The router consumes the ingest queue and publishes one delivery message per matching pipeline to the delivery queue, again confirmed, before it acks the event. A message the router cannot handle is retried 3 times in place, then moved to the ingest DLQ (listed aslegacyby the DLQ endpoint). - Deliver. The delivery worker consumes the delivery queue in batches and settles every message one way before acking it:
| Outcome | When | What happens |
|---|---|---|
| delivered | transformer and sink succeeded | the pipeline is marked done for the event (a duplicate is skipped) |
| deferred | pipeline rate limit, tenant in-flight cap, full bulkhead | parked in the 5 s tier, no attempt spent |
| retried | retryable failure (5xx, timeout, open circuit, missing secretRef) | parked in the next delay tier: 5 s, 30 s, 2 min, 10 min, 30 min, 1 h, then 3 h |
| dead-lettered | poison (4xx, malformed transformer output), maxAttempts used up, maxAge passed, pipeline removed | one entry in labs64-audit-dlq.<tenant> with reason, attempts and last error |
| quarantined | unparseable event, or the tenant was removed or disabled after ingest | kept in labs64-audit-quarantine.quarantine with an x-quarantine-reason header; never delivered to a sink, not replayed by the DLQ endpoint |
On RabbitMQ delays are queues with a fixed TTL that dead-letter back to the delivery exchange (labs64-audit-delay.<seconds>s), so no broker plugin is needed and it works the same on Amazon MQ. On SQS a delay is the message’s own delivery delay. Per pipeline, in the tenant file:
pipelines:
- name: archive
retry:
maxAttempts: 20 # default auditflow.delivery.retry.max-attempts
maxAge: 24h # default auditflow.delivery.retry.max-age; also PT24H, 30m, 2d
batch:
enabled: true # off by default
maxSize: 100 # upper bound of events per sink call, 1-1000
Sink batching. With batch.enabled the worker groups the deliveries it received for a pipeline of a tenant and calls the sink’s POST /sink/<id>/batch once per group. The sink answers per event, and each event is settled on its own (delivered, retried, deferred or dead-lettered), so a refused event never fails the others.
batch.enabledis the switch;batch.maxSizealone does nothing.maxSizeis 1 to 1000 (default 100); a value outside that range rejects the tenant file.- The size that is reached. A group is cut from ONE batch a consumer received, so it never holds more than
auditflow.delivery.batch-size(50) events, and fewer when that batch mixes pipelines or tenants.GET /actuator/pipelines/<tenantId>shows it asbatch.effectiveMaxSize. For larger sink batches raiseAUDITFLOW_DELIVERY_BATCH_SIZEtogether withmaxSize. - When a batch closes. When it is full, or after
auditflow.delivery.batch-receive-timeout(1 s) without new messages, or, whenauditflow.delivery.batch-windowis set (off by default), once that long has passed since its first message (checked between messages, so up to one receive timeout later). Under light load batches are small and add up to 1 s of latency. The window trades latency for fewer, larger sink objects, for EVERY pipeline (they share one queue); the consumer also waits up to the receive timeout on shutdown, so keep that one short (seconds) and the window under a few minutes. Messages of a batch that is still gathering stay unacked, so a stopping pod hands them back to the broker; nothing is lost, and delivery marks keep a repeat from duplicating work. - Never across tenants. Two tenants with a pipeline of the same name get separate calls.
- One slot, one call. A batch takes one slot of the tenant’s in-flight cap and is one call for the bulkhead. The call has the same 10 s response timeout as a single event.
- Retries. Only the failed events of a batch are retried, each on its own schedule. They may come back grouped with other events, so a sink cannot count on seeing the same group twice.
- Fallback sink. The events that failed with a retryable error go to the fallback in one call.
What a sink does with a batch:
| Sink | One batch becomes | A repeat after a lost answer |
|---|---|---|
aws_s3_sink, gcs_sink, azure_blob_sink | one JSON Lines object per partition folder | the same group overwrites its object; a different grouping can repeat an event in a second object |
clickhouse_sink | one INSERT ... FORMAT JSONEachRow | merged away by the ReplacingMergeTree key of the example schema |
postgres_sink | one multi-row INSERT in one transaction | skipped with a unique index on eventId, otherwise a second row |
snowflake_sink | one connection and one multi-row INSERT | a second row |
opensearch_sink | one _bulk request, an outcome per document | indexed again (ids are generated) |
aws_cloudwatch_sink | one PutLogEvents call per 10,000 events or 1 MB; the group and stream are checked once | stored again |
datadog_sink | one request per 1000 entries or 5 MB | stored again |
splunk_sink | one HEC request | indexed again |
loki_sink | one push, equal label sets merged and ordered by time | dropped by Loki when stream, time and line are equal |
logging_sink, webhook_sink, syslog_sink, netlicensing_sink | the events through process, up to 8 at a time | as for single events |
For every destination that stores a repeat, readers de-duplicate by eventId. Batching pays off most for ClickHouse and Snowflake (one insert instead of one per event), then for the object stores (one object instead of thousands of small ones). For a sink in the last row it only saves calls between the backend and the sink service; keep maxSize small there, so that the events fit into the 10 s call (maxSize x the destination’s latency / 8).
Throughput. An event mostly waits on the transformer and sink, so concurrency, not CPU, sets the rate (CPU-based autoscaling does not react to a backlog).
| Setting | Default | Purpose |
|---|---|---|
AUDITFLOW_DELIVERY_CONCURRENCY (env) / auditflow.delivery.concurrency | 4 | Delivery consumers per pod |
AUDITFLOW_DELIVERY_BATCH_SIZE (env) / auditflow.delivery.batch-size | 50 | Messages a consumer takes at once (also its prefetch) |
auditflow.delivery.unit-concurrency | 16 | Deliveries of one received batch run in parallel |
auditflow.circuitbreaker.bulkhead-max-concurrent-calls | 128 | Concurrent calls per transformer/sink target; keep >= concurrency x unit-concurrency |
tenants.consumer.max-in-flight-per-tenant | 32 | One tenant’s deliveries at once per pod (fairness); over it, a delivery waits up to max-wait-millis (2 s), then is deferred |
AUDITFLOW_CONSUMER_CONCURRENCY (env) | 8 | Router threads on the ingest queue (light work) |
AUDITFLOW_BROKER_QUEUE_TYPE (env) / auditflow.broker.queue-type | classic | RabbitMQ: quorum for replicated queues on a multi-node broker |
A local run (one pod, single-process transformer and sink) delivered 3,000 events, half of them through a batching pipeline, in about 6 s end to end. Fairness between tenants comes first from the ingest quota (rateLimitPerSec/burst), then from the in-flight cap.
Metrics: auditflow.delivery.outcomes{tenant,pipeline,outcome} (delivered, deferred, retried, duplicate), auditflow.delivery.deadlettered{tenant,pipeline,reason}, auditflow.delivery.batch.size, auditflow.pipeline.duration{pipeline,outcome,mode}, auditflow.delivery.latency{tenant,pipeline} (enqueued to delivered, retries included; histogram buckets 1 s … 3 h). Alert on deadlettered > 0 and on the depth of the tenant DLQs; a delivery-freshness SLO reads the 5-minute bucket: auditflow_delivery_latency_seconds_bucket{le="300.0"} over auditflow_delivery_latency_seconds_count is the share delivered within 5 minutes. Every pipeline’s ENQUEUED (auditflow.pipeline.outcomes) ends as delivered, duplicate or dead-lettered: a gap between the two over a few hours is deliveries lost without an error.