Troubleshooting
Backend won’t start
Symptom: APPLICATION FAILED TO START in logs.
- Check the broker is reachable (on SQS:
/actuator/healthshows thesqscomponent, and the backend log names anAccessDeniedor a missing queue). RabbitMQ:docker logs auditflow-rabbitmq | tail -5 curl -s http://localhost:15673/api/overview | head -
Check credentials:
RABBITMQ_USERNAMEandRABBITMQ_PASSWORDmust be set (no defaults by design). - Check port conflicts:
lsof -i :8080 -i :8081 -i :8082 -i :5673 -i :15673
Sink returns 422 Unprocessable Content
Symptom: Backend logs show 422 Unprocessable Content from POST http://sink:8082/sink/{name}.
This usually means the request body format doesn’t match what the sink expects. The backend sends:
{"event_data": {...}, "properties": {...}}
The sink’s /sink/{name} endpoint must accept this structure as a single JSON body.
Sink returns 500 Internal Server Error
Symptom: Sink logs show An unexpected error occurred in sink endpoint.
Check the full traceback in sink logs:
just log sink
Common causes:
- Missing Python package in
requirements.txt - Sink module has a bug in its
process()function - External service (e.g., S3, Loki) is unreachable
Transformer returns 500
Same approach — check just log transformer for the full traceback.
Events not being consumed
Symptom: Events published but sink logs show nothing.
- Check the consumer is subscribed:
just log backend | grep "subscriber" - Check RabbitMQ queue depth:
- Open http://localhost:15673
- Go to Queues →
labs64-audit-topic.labs64.io-auditflow - If Ready count keeps growing, the consumer is failing
- Check for circuit breaker open state:
curl -s http://localhost:8080/actuator/metrics | grep circuitbreaker
Dead Letter Queue filling up
A delivery lands in its tenant’s DLQ (labs64-audit-dlq.<tenant>) when it is poison, used up its pipeline’s retry.maxAttempts, passed retry.maxAge, or its pipeline was removed. Each entry is one pipeline of one event, with x-auditflow-dlq-reason and x-auditflow-last-error headers. Entries from before per-pipeline delivery, and events whose routing failed, sit in the legacy shared queue labs64-audit-topic.labs64.io-auditflow.dlq; the same endpoint covers both. Every operation requires a {tenantId} selector (use _platform for tenantless events); there is no un-scoped /actuator/dlq path. Check the DLQ for a tenant (byReason tells poison from outages):
curl -s http://localhost:8080/actuator/dlq/<tenantId> | python3 -m json.tool
Once the cause is fixed, replay that tenant’s entries (or one pipeline’s):
curl -X POST http://localhost:8080/actuator/dlq/<tenantId> | python3 -m json.tool
curl -X POST http://localhost:8080/actuator/dlq/<tenantId> -H 'Content-Type: application/json' \
-d '{"pipeline":"archive"}' | python3 -m json.tool
Purge (discard, don’t replay) that tenant’s entries, or one pipeline’s — irreversible; every discarded entry is logged with its event id:
curl -X DELETE http://localhost:8080/actuator/dlq/<tenantId> | python3 -m json.tool
curl -X DELETE "http://localhost:8080/actuator/dlq/<tenantId>?pipeline=archive" | python3 -m json.tool
Container healthcheck failing
The healthchecks use wget to hit /health on each service. If a container shows unhealthy:
- Check if the process is running:
docker exec auditflow-transformer ps aux - Test the health endpoint from inside the container:
docker exec auditflow-transformer wget -qO- http://127.0.0.1:8081/health - Check for import errors (e.g., missing
health.pyin the image):docker exec auditflow-transformer ls /home/l64user/health.py
Idempotency: events being dropped unexpectedly
The idempotency guard uses eventId as the dedup key. If you’re sending events without an eventId, they won’t be deduplicated — but they also won’t be protected against redelivery duplicates.
Check the current claim TTL (default 5 minutes):
curl -s http://localhost:8080/actuator/metrics | grep deduplicated
Publish returns 400 “eventTime … is in the future” / “too old”
eventTime is when the action happened, and ingest only accepts a plausible one: not later than the receipt time plus auditflow.event-time.max-future (5 minutes of clock skew) and not older than max-age (1095 days). A future value means the publisher’s clock is ahead (or it sent a deadline, which belongs in extra); a too-old one is a backfill beyond what the sinks keep. Fix the clock, or omit eventTime to use the receipt time. A backfill that is legitimate and older than three years needs max-age raised and a sink retention that keeps it (ClickHouse expires rows 1095 days after event_time). See Time fields.