sqs-mini – Event-Driven Pipeline

Problem / Context

Message queues, retries, and idempotent consumers are patterns I kept seeing in backend architecture discussions but hadn't built from scratch. sqs-mini is my working implementation of the full flow: a Minimal API publishes a message to SQS, a Worker Service picks it up, processes it once (SQLite keeps track of seen message IDs), and anything that fails enough times lands in the DLQ. Pulumi provisions the whole AWS setup in C# — same language as the rest of the project.

Solution / Architecture

The project is composed of two .NET applications: a Minimal API (`SessionApi`) acting as the message producer, and a Worker Service (`BillingWorker`) acting as the consumer. The infrastructure – an SQS Standard Queue and its DLQ – is provisioned using Pulumi in C#. The worker uses long polling to efficiently receive messages, processes them idempotently using a SQLite store to track already-handled message IDs, and lets SQS handle retries and DLQ routing for unprocessable messages. Runtime configuration is environment-based to keep credentials and queue URLs out of source code.

Goals

• Demonstrate a realistic producer → queue → consumer flow
• Provision AWS infrastructure declaratively with Pulumi IaC in C#
• Implement long polling in a .NET Worker Service
• Handle idempotency with a SQLite-backed message ID store
• Show retries and dead-letter queue behavior clearly

Challenges

• Designing an idempotency store that is lightweight but reliable for the demo use case
• Configuring Pulumi to provision SQS queues with the correct DLQ linkage and retry policies
• Handling environment-based config cleanly without leaking secrets into source code

Key Technical Decisions

• Pulumi in C# for IaC – keeps the entire stack in one language the team knows
• SQLite for the idempotency store – zero infrastructure cost for a demo, easy to swap for Redis or DynamoDB in production
• Worker Service pattern – the idiomatic .NET approach for long-running background consumers
• Minimal API for the producer – lightweight and fast to stand up

Results / Impact

• A working end-to-end event-driven pipeline: API publishes → SQS delivers → Worker consumes → DLQ catches failures
• Infrastructure fully reproducible with a single Pulumi command
• Demonstrated understanding of distributed systems patterns: idempotency, retries, DLQ, at-least-once delivery

What I Would Improve Next

• Replace SQLite idempotency store with DynamoDB for a fully cloud-native solution
• Add structured logging and observability with CloudWatch
• Extend to FIFO queue for strict ordering guarantees