Message Queues

Why a queue

A message broker is middleware that receives messages from producers, stores them durably, and delivers them to consumers — decoupling the two. Putting a broker (NATS — nats.io — and Kafka — kafka.apache.org — are popular cloud-native ones; RabbitMQ and cloud SQS are others) between services turns a synchronous, fragile call into asynchronous, decoupled communication:

• Decoupling — the producer publishes and moves on; it doesn’t wait for, or depend on, the consumer being up.
• Buffering — the queue absorbs bursts so the consumer works at a steady rate.
• Independent scaling & failure — add consumers to drain faster; either side can restart without failing the other.

The cost is added latency and eventual consistency — the work happens soon, not now.

Queue vs pub/sub

graph TD
subgraph Queue["work queue (point-to-point)"]
  P1["producer"] --> Q1["queue"]
  Q1 --> C1["consumer A"]
  Q1 --> C2["consumer B"]
  Q1 -.one msg → one consumer.-> C1
end
subgraph PubSub["pub/sub (topic)"]
  P2["producer"] --> T["topic"]
  T --> S1["subscriber X (all get it)"]
  T --> S2["subscriber Y (all get it)"]
end

• Work queue — each message goes to one of the competing consumers (distribute work).
• Pub/sub — each message goes to every subscriber (broadcast an event). Kafka consumer groups / NATS queue groups give you both: fan-out across groups, load-balance within one.

See it: pub/sub with ack and redelivery

This runs here — a channel-based broker fans a message out to subscribers, and an unacked (failed) delivery is redelivered, illustrating at-least-once. Output is deterministic:

package main

import "fmt"

type msg struct {
id   int
body string
}

// processWithRetry models at-least-once: redeliver until acked (or give up).
func processWithRetry(m msg, handle func(msg) bool) {
for attempt := 1; attempt <= 3; attempt++ {
	if handle(m) { // returns true on successful ack
		fmt.Printf("msg %d acked on attempt %d\n", m.id, attempt)
		return
	}
	fmt.Printf("msg %d nacked, redelivering (attempt %d)\n", m.id, attempt)
}
fmt.Printf("msg %d → dead-letter queue\n", m.id)
}

func main() {
// A handler that fails msg 2 once, then succeeds (transient error).
seen := map[int]int{}
handle := func(m msg) bool {
	seen[m.id]++
	return !(m.id == 2 && seen[m.id] == 1)
}

for _, m := range []msg{{1, "a"}, {2, "b"}, {3, "c"}} {
	processWithRetry(m, handle)
}
}

Message 2 fails once and is redelivered, then acked — at-least-once delivery in miniature. A message that fails repeatedly lands in a dead-letter queue for inspection rather than blocking the stream. The real client is similar (fenced — NATS here):

// Fenced: github.com/nats-io/nats.go — a queue group load-balances delivery.
nc, _ := nats.Connect(nats.DefaultURL)
nc.QueueSubscribe("orders", "workers", func(m *nats.Msg) {
	if err := process(m.Data); err != nil {
		return // no ack → redelivered (with JetStream)
	}
	m.Ack()
})

Delivery guarantees

You choose where to ack, and that picks your guarantee:

• At-most-once — ack on receipt. Fast, but a crash mid-processing loses the message.
• At-least-once — ack after processing. No loss, but a crash before ack causes a duplicate redelivery. This is the practical default.
• Exactly-once — effectively unachievable across a network; you simulate it with at-least-once + idempotent consumers (dedupe by an idempotency key), covered in the outbox pattern.

See also

• Event-driven architecture — designing systems around messages.
• The outbox pattern — reliable publishing and idempotency.
• Resilience patterns — backpressure, retries, and dead-lettering.
• Rate limiting (concurrency) — bounding producers and consumers.

Next: building whole systems around events — event-driven architecture.

Related topics