How a message queue decouples two services
A message queue is a waiting line for work between two services. The sender drops a message in and moves on, so neither side has to wait for the other.
The short version
The producer puts a message on the queue and goes back to its own work. A consumer takes messages off the queue when it is ready, and the two need not run at the same time or on the same machine. Messages wait in the queue until someone is free, and adding more consumers shares the load between them. Think of a post box: the sender drops the letter and leaves, and the recipient collects it later.
Step by step
- The producer sends a message to the queue and carries on.
- The queue holds the message until a consumer is ready.
- A consumer receives it and does the work. In Amazon SQS the message stays on the queue meanwhile, hidden from other consumers.
- When the work is done, the consumer confirms (RabbitMQ calls it an ack, SQS deletes the message) and the message is gone.
- If the consumer dies before confirming, RabbitMQ puts the message back for another consumer to take.
| Without a queue | With a queue | |
|---|---|---|
| Consumer is slow or down | The producer waits or fails | Messages wait in the queue |
| A burst of work arrives | The consumer takes all of it at once | Consumers work through the backlog at their own pace |
| Consumer crashes mid-job | The producer has to notice and send it again | The message returns and another consumer takes it |
Why it matters
A queue soaks up bursts of work, and a crashed consumer loses nothing. The price is that standard SQS queues deliver at least once, so the same message can arrive twice and the consumer has to cope with that. My take: the result also arrives later instead of immediately, and there is one more part to run.
Sources
- What is Amazon Simple Queue Service? (AWS): queues that decouple distributed systems, the producer and consumer roles, and the message lifecycle from send to delete
- Amazon SQS at-least-once delivery (AWS): standard queues deliver at least once, a message can be received more than once, and applications should handle that
- RabbitMQ tutorial: work queues: the producer is not blocked, adding workers shares the load, a worker sends an ack when done, and an unacknowledged message is requeued for another consumer