Project 05
Event-Driven Ingestion Pipeline
The layer that feeds everything above, with lineage on every record
New or updated documents raise events. Consumers enrich metadata, run PII detection and redaction before anything reaches an LLM, chunk, embed and upsert into the index. Every record carries its source, version and the full chain of transformations applied — so any answer in the knowledge base can be traced back to the document revision it came from.
Try it
How it works
- 01Event
S3 put or manual submit raises an EventBridge event.
- 02Queue
SQS with a dead-letter queue — retries are free, failures are visible.
- 03Redact
PII scan on names, emails, phone numbers and financial identifiers before any external call.
- 04Transform
Chunk, enrich, embed, upsert — idempotent by content hash.
- 05Record
Lineage row per document version: source, transformations, output index IDs.
Infrastructure & running cost
| Service | Role | Cost |
|---|---|---|
| EventBridge + SQS | Event backbone. MSK Serverless would be ~$100/mo | ~$0.01/mo |
| Lambda | Consumers, scaled by queue depth | Free tier |
| DynamoDB | Lineage and catalogue tables | Free tier |
Serverless event backbone in place of Kafka — same patterns, 1/1000th the cost.