Message Queue Patterns for Reliable Delivery

In high-scale production systems processing thousands of messages daily, "probably delivered" isn't good enough. When dealing with business-critical operations like payment processing or document handling, reliability is paramount. Here's how to approach reliable message delivery using RabbitMQ.

The Challenge

Distributed systems have fundamental challenges that make "exactly-once" delivery surprisingly hard:

1. Networks fail
2. Processes crash at inconvenient times
3. Messages can be delivered but not acknowledged
4. Retries can cause duplicate processing

The goal isn't to eliminate these problems. It's to handle them gracefully.

At-Least-Once vs Exactly-Once

Most message systems guarantee at-least-once delivery. You'll get the message, but maybe more than once. This shifts the complexity to consumers, who must be idempotent.

True exactly-once delivery requires coordination between the message system and your application state. This is expensive and often unnecessary. We use exactly-once semantics (the appearance of exactly-once from the application's perspective) instead.

The Outbox Pattern

Our most important pattern is the transactional outbox:

async def process_order(order_data):
    async with database.transaction():
        # Save order
        order = await orders.create(order_data)
        
        # Write to outbox in same transaction
        await outbox.create({
            'aggregate_id': order.id,
            'event_type': 'order_created',
            'payload': serialize(order),
            'created_at': datetime.now()
        })
    
    # Outbox processor publishes to queue separately

The key insight: the database transaction guarantees that the order and the outbox entry either both exist or neither exists. A separate process reads the outbox and publishes to RabbitMQ.

This decouples the business logic from message publishing and handles crashes gracefully.

Idempotent Consumers

Despite our best efforts, consumers will sometimes receive duplicates. Every consumer must handle this:

async def handle_message(message):
    # Check if we've already processed this message
    message_id = message.properties.message_id
    
    if await processed_messages.exists(message_id):
        logger.info(f"Skipping duplicate: {message_id}")
        return
    
    async with database.transaction():
        # Process the message
        await do_business_logic(message.body)
        
        # Record that we processed it
        await processed_messages.create({
            'message_id': message_id,
            'processed_at': datetime.now()
        })

Store message IDs in your database, not in memory. Processes restart, and you need durability.

Dead Letter Queues

Messages that repeatedly fail need special handling. We use a tiered retry approach:

1. Immediate retry: For transient failures (network blips)
2. Delayed retry: Exponential backoff for recoverable failures
3. Dead letter queue: For messages that fail after max retries

class MessageHandler:
    max_retries = 5
    
    async def handle(self, message):
        retry_count = message.headers.get('x-retry-count', 0)
        
        try:
            await self.process(message)
        except TransientError:
            if retry_count < self.max_retries:
                await self.retry_with_delay(message, retry_count + 1)
            else:
                await self.send_to_dlq(message)
        except PermanentError:
            await self.send_to_dlq(message)

We monitor the DLQ and alert when messages land there. Someone needs to investigate why messages are failing permanently.

Atomic Job Claiming

For long-running jobs, we use atomic claiming to prevent duplicate processing:

UPDATE jobs 
SET status = 'processing', 
    worker_id = $1,
    started_at = NOW()
WHERE id = $2 
  AND status = 'pending'
RETURNING *;

If the update returns a row, you own it. If not, another worker already claimed it. Simple and reliable.

Circuit Breaker Integration

Message consumers can also trip circuit breakers:

class ConsumerWithCircuitBreaker:
    def __init__(self):
        self.circuit = CircuitBreaker(
            failure_threshold=5,
            recovery_timeout=timedelta(seconds=30)
        )
    
    async def handle(self, message):
        if not self.circuit.can_execute():
            # Requeue for later processing
            await self.requeue_with_delay(message)
            return
        
        try:
            await self.process(message)
            self.circuit.record_success()
        except Exception as e:
            self.circuit.record_failure()
            raise

This prevents consumers from hammering a failing downstream service.

Monitoring and Observability

We track several metrics:

• Queue depth: How many messages are waiting
• Processing time: P50, P95, P99 latencies
• Failure rate: Percentage of messages that fail
• DLQ size: Messages that exhausted retries
• Consumer lag: How far behind are we

Alerts trigger when:

• Queue depth exceeds threshold for > 5 minutes
• DLQ receives any messages
• Processing time degrades significantly

Ordering Guarantees

Sometimes message order matters. Options:

1. Single consumer: Process all messages for a category sequentially
2. Partition by key: Route related messages to the same queue
3. Sequence numbers: Consumer reorders if needed

We mostly avoid strict ordering requirements. They limit scalability and add complexity.

Lessons Learned

Design for failure: Every message will eventually fail. Have a plan.

Idempotency is non-negotiable: If your consumer can't handle duplicates, you'll have data corruption.

Monitor the DLQ: An empty DLQ means either everything works or nobody's looking.

Test the sad path: Simulate network failures, slow consumers, poison messages. Production will throw these at you.

Keep messages small: Put large payloads in object storage and pass references.

Reliable messaging isn't about preventing failures. It's about handling them gracefully. Get the patterns right, and your system can survive the inevitable chaos of distributed computing.

Sources and further reading

Where the guarantees above are actually specified, and where to go deeper.

• RabbitMQ: Reliability guide. Publisher confirms, consumer acknowledgements and what the broker does and does not promise. The section on why acknowledgements must come after the work is the one people skip.
• RabbitMQ: Dead letter exchanges. How messages end up in a dead letter queue, and the TTL trick that turns one into a delayed retry.
• Pub/Sub: Exactly-once delivery. A precise statement of what a broker can guarantee, and the narrow conditions under which it holds. Useful for calibrating how much you should trust the phrase.
• SQS: Visibility timeout. The mechanism behind redelivery, and the one setting that decides how long a crashed consumer blocks a message.
• Transactional outbox on microservices.io. The pattern that makes a database write and its announcement atomic.
• Timeouts, retries, and idempotency by Marc Brooker. Short and precise on why a retry is only safe when the operation is idempotent.
• Addressing cascading failures from the Google SRE book, for what happens when every consumer retries into a struggling dependency at once.

Continue learning

If delivery semantics now make sense, the next thing worth understanding properly is the outbox pattern, because it closes the gap between your database and your broker that at-least-once alone cannot. After that, idempotency key design, which is what makes redelivery harmless rather than merely survivable. Then consumer group and partition mechanics, which is where ordering guarantees actually come from.

FAQ

Does any broker really give exactly-once delivery? Not end to end. What brokers offer is exactly-once processing under specific conditions, usually within their own system. Across a network boundary, plan for at-least-once and make the work idempotent.

Do I need an outbox if I already have retries? Yes, if a database write and its message must not diverge. Retries fix a failed send; they do not fix a message sent for a transaction that later rolled back.

When should a message go to a dead letter queue? When redelivery will not help, meaning malformed payloads and permanent failures. A rate limit is not that, and sending it to the dead letter queue turns a delay into lost work.