sqs-mini – Event-Driven Pipeline
Backend project showing an event-driven pipeline: a Minimal API publishes messages to AWS SQS, a .NET Worker Service consumes them with long polling and idempotent processing, and failures route to a DLQ. Infrastructure is provisioned in C# via Pulumi.
Tech Stack
C# · .NET · AWS SQS · Pulumi · Minimal API · Worker Service · SQLite
Problem / Context
Message queues, retries, and idempotent consumers are patterns I kept seeing in backend discussions but hadn't built from scratch. sqs-mini is my working implementation of the full flow: a Minimal API publishes to SQS, a Worker Service picks it up, processes it once (SQLite tracks seen message IDs), and anything that fails enough times goes to the DLQ. Pulumi provisions the AWS setup in C#, the same language as the rest of the project.
Solution / Architecture
The project is composed of two .NET applications: a Minimal API (`SessionApi`) acting as the message producer, and a Worker Service (`BillingWorker`) acting as the consumer. The infrastructure – an SQS Standard Queue and its DLQ – is provisioned using Pulumi in C#. The worker uses long polling to efficiently receive messages, processes them idempotently using a SQLite store to track already-handled message IDs, and lets SQS handle retries and DLQ routing for unprocessable messages. Runtime configuration is environment-based to keep credentials and queue URLs out of source code.
Goals
- Demonstrate a realistic producer → queue → consumer flow
- Provision AWS infrastructure declaratively with Pulumi IaC in C#
- Implement long polling in a .NET Worker Service
- Handle idempotency with a SQLite-backed message ID store
- Show retries and dead-letter queue behavior clearly
Challenges
- Designing an idempotency store that is lightweight but reliable for the demo use case
- Configuring Pulumi to provision SQS queues with the correct DLQ linkage and retry policies
- Handling environment-based config cleanly without leaking secrets into source code
Key Technical Decisions
- Pulumi in C# for IaC – keeps the entire stack in one language the team knows
- SQLite for the idempotency store – zero infrastructure cost for a demo, easy to swap for Redis or DynamoDB in production
- Worker Service pattern – the idiomatic .NET approach for long-running background consumers
- Minimal API for the producer – lightweight and fast to stand up
Results / Impact
- A working event-driven pipeline: API publishes to SQS, Worker consumes, DLQ catches failures
- Infrastructure fully reproducible with a single Pulumi command
- Practical understanding of distributed systems patterns: idempotency, retries, DLQ, at-least-once delivery
What I Would Improve Next
- Replace SQLite idempotency store with DynamoDB for a fully cloud-native solution
- Add structured logging and observability with CloudWatch
- Extend to FIFO queue for strict ordering guarantees