Message queues and background jobs

Don't make the user wait

When a customer places an order, several things should happen: save the order, charge the card, send a confirmation email, update analytics, notify the warehouse, generate an invoice PDF. Doing all of that inside the HTTP request makes it slow and fragile — if the email server is down, should the order fail?

The answer is to do only what's essential now, and hand the rest to background processing via a queue.

         POST /orders                 enqueue                    dequeue
Client ────────────────▶ API ─────────────────────▶ [ Queue ] ─────────────▶ Worker
       ◀──── 202 Accepted ──                           ▲ ▲ ▲               (sends email,
           (or 201 + order id)                       buffered                makes PDF…)

Queues: producers and consumers

A message queue (RabbitMQ, Azure Service Bus, Amazon SQS, Kafka, or Redis streams) stores messages until a consumer takes them:

• The producer (your API) publishes a message — e.g. { "orderId": 42 } — and moves on.
• One or more consumers (workers) receive messages, process them, then acknowledge them.
• If a worker crashes before acknowledging, the message becomes visible again and another worker retries it.

Benefits:

• Responsiveness — requests return immediately.
• Decoupling — the API doesn't need the email service to be up right now.
• Load leveling — a burst of 10,000 orders queues up and workers process them at a steady pace.
• Scalability — add more workers to drain the queue faster.

Queue vs publish/subscribe

Events let new features subscribe to existing events without changing the producer — a key idea in event-driven architecture.

Delivery guarantees and idempotency

Networks and processes fail, so brokers typically guarantee at-least-once delivery: a message may arrive more than once. Consumers must therefore be idempotent:

async Task HandleAsync(OrderPlaced message)
{
    if (await db.ProcessedMessages.AnyAsync(m => m.Id == message.MessageId))
        return;                                         // already handled — skip
 
    await emailSender.SendConfirmationAsync(message.OrderId);
    db.ProcessedMessages.Add(new ProcessedMessage(message.MessageId));
    await db.SaveChangesAsync();
}

Retries and dead letters

Transient failures (a timeout, a deadlock) deserve a retry — usually with a delay that grows each time. But a poison message that always fails (malformed data, a bug) would be retried forever and block others. After N attempts, brokers move it to a dead-letter queue for humans to inspect and replay.

The dual-write problem and the outbox

Saving an order to the database and publishing a message are two separate operations. If the app crashes between them, you get an order without an event (or an event without an order). The transactional outbox pattern fixes this: write the message to an outbox table in the same database transaction as the order, and let a background process publish outbox rows to the broker.

Background work in .NET

• BackgroundService / IHostedService — long-running loops inside your app (e.g. consuming a queue, cleaning up expired data).
• Channel<T> — an in-process queue between API and background service (Track 6) — simple, but lost on restart.
• Hangfire, Quartz.NET — scheduled and recurring jobs with persistence and dashboards.
• MassTransit, NServiceBus, Wolverine — messaging frameworks on top of RabbitMQ / Azure Service Bus, handling retries, outbox and sagas.

public class InvoiceWorker(Channel<int> orders, IServiceScopeFactory scopes) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        await foreach (var orderId in orders.Reader.ReadAllAsync(stoppingToken))
        {
            using var scope = scopes.CreateScope();
            var invoices = scope.ServiceProvider.GetRequiredService<InvoiceService>();
            await invoices.GenerateAsync(orderId, stoppingToken);
        }
    }
}

Key takeaways

• Keep requests fast: do essential work synchronously, queue the rest, return 202 Accepted when appropriate.
• Queues decouple producers from consumers, level load and let you scale workers independently.
• Queues deliver commands to one consumer; pub/sub delivers events to all subscribers.
• Expect at-least-once delivery: idempotent consumers, retries with backoff, dead-letter queues, and the outbox for reliable publishing.