Message Brokers Compared: Kafka vs RabbitMQ vs SQS

Quick Answer

Kafka, RabbitMQ, and SQS all move messages between services, but they're built on fundamentally different models: Kafka is a distributed, partitioned log that keeps messages around regardless of who's read them; RabbitMQ is a smart broker that routes messages and deletes them once a consumer acknowledges; SQS is a fully managed queue with zero broker infrastructure to run. Pick based on your access pattern — replay and high-throughput streaming favor Kafka, flexible routing favors RabbitMQ, and zero-ops simplicity on AWS favors SQS.

Three Fundamentally Different Architectures, Not Three Flavors of One Thing

Kafka is log-based. A topic is split into partitions, and each partition is an ordered, append-only, immutable log written sequentially to disk. The broker's job is deliberately narrow: append new records to the end of a partition, and serve a byte range starting at whatever offset a consumer asks for. Reading never deletes anything, which is why multiple, completely independent consumer groups can read the exact same messages at their own pace without stepping on each other — Kafka records a message and keeps it until its retention period expires, regardless of how many readers have already seen it.

RabbitMQ is queue-based, and it's genuinely a "smart broker" in a way Kafka isn't trying to be. A producer publishes to an exchange, which routes a copy of the message to whichever queues are bound to it according to routing rules, and a consumer pulls (or gets pushed) a message off a queue. The moment that consumer acknowledges it, the broker deletes the message — it's gone, for every consumer, permanently.

SQS isn't broker software at all — it's a managed queue service where AWS owns and scales the underlying infrastructure. A message sits in a single named queue until some consumer receives and explicitly deletes it, or until its retention period (configurable up to 14 days) expires on its own. There's no cluster to provision, no partitions to plan, and no exchange topology to design — the entire operational surface is the queue itself.

Ordering Guarantees: What Each One Actually Promises

Kafka guarantees ordering only within a single partition. Messages published with the same key are always routed to the same partition via a hash of that key, so per-entity ordering — every event for one specific user ID, for instance — is preserved reliably. Across partitions of the same topic, though, there's no ordering guarantee at all; that's the price of parallelism, and it's a deliberate trade-off, not an oversight.

RabbitMQ guarantees ordering within a single queue, but only as long as there's exactly one consumer reading it. The instant you add competing consumers to scale throughput, ordering stops being guaranteed — a slower consumer can still be working through an earlier message while a faster one has already grabbed, processed, and acknowledged a later one.

SQS splits this cleanly by queue type. Standard queues offer what AWS documentation calls "loose FIFO" — a best-effort attempt at order, with no guarantee, because the queue is deliberately built on a highly distributed backend for near-unlimited throughput. FIFO queues, in contrast, preserve strict ordering per message group and provide effectively exactly-once processing — at a real, documented throughput cost covered next.

Throughput, Retention, and What Happens After a Message Is Read

The numbers here are worth having straight from AWS's own documentation rather than a rough guess. SQS FIFO queues support up to 300 send, receive, or delete operations per second without batching, or up to 3,000 messages per second when batching requests — and if that's not enough, enabling High Throughput Mode raises the ceiling much further, into the tens of thousands of messages per second, by spreading load across more partitions per message group ID. SQS Standard queues carry no such ceiling at all, because they don't need to preserve strict per-message ordering to hit that scale.

Kafka's throughput advantage comes from a genuinely different place: sequential disk writes. An append-only log is about the cheapest possible disk access pattern there is, and combined with consumer-side batching and zero-copy network sends, Kafka is built to sustain very high write and read volumes. Its retention is time- or size-based and completely independent of who's consumed what — which is exactly why Kafka topics are routinely used as a genuine system of record you can replay from months later, not just a transient pipe between two services.

RabbitMQ and SQS both share a trait Kafka doesn't: once a message is consumed and acknowledged (RabbitMQ) or received-and-deleted (SQS), it's gone for that queue. Neither gives you Kafka's default ability for a second, independent consumer group to come along later and re-read history from an earlier offset — RabbitMQ's newer Streams feature exists specifically to add log-like replay semantics on top, precisely because the base queue model doesn't have it.

How Messages Actually Get Distributed to Consumers

RabbitMQ's real strength is routing flexibility, and it's genuinely more capable here than either alternative. A direct exchange routes on an exact match between a message's routing key and a queue's binding key. A fanout exchange broadcasts a copy to every bound queue and ignores the routing key entirely. A topic exchange matches routing keys against wildcard patterns, letting one publisher fan out to very different subscriber sets based on a structured key like order.payment.failed. A headers exchange routes on message header values instead of the routing key at all. That range makes RabbitMQ genuinely good at "route this kind of message to these specific downstream services" topologies where the producer shouldn't need to know who's listening.

Kafka's consumption model is comparatively simple, and that simplicity is the point: a partition is the unit of parallelism, and within one consumer group, each partition is owned by exactly one consumer at a time. Scaling a consumer group out, up to the partition count, gives you parallel processing with ordering preserved per partition — but adding consumers beyond the partition count just leaves the extras idle, which is a real capacity-planning decision you make when you choose how many partitions a topic gets.

SQS keeps it simpler still: one queue, with competing consumers polling it directly (short or long polling). A visibility timeout hides an in-flight message from every other consumer until it's deleted or that timeout expires; if a consumer crashes mid-processing, the message simply reappears for someone else to pick up. That's SQS's version of at-least-once delivery, and it requires no exchange, routing key, or topic configuration at all.

The Operational Reality: What You're Actually Signing Up to Run

Running Kafka means running a cluster of brokers — either yourself or via a managed offering like Confluent Cloud or Amazon MSK. Older Kafka deployments coordinate that cluster through ZooKeeper; modern Kafka's KRaft mode removes that external dependency and lets brokers manage consensus themselves. Either way, you're planning partition counts and replication factors upfront, because repartitioning a live, high-traffic topic later is disruptive rather than a routine config change.

RabbitMQ, an Erlang-based broker, is comparatively lightweight to operate — a single node is genuinely simple to stand up, clustering for availability is well documented, and its plugin ecosystem (delayed messages, priority queues, the Streams feature mentioned earlier) covers a lot of ground that would otherwise require external tooling.

SQS asks for none of that: no broker to patch, no cluster to size, no version to upgrade. The trade is full commitment to AWS's specific API, quota model, and the exact guarantees of whichever queue type you choose — it's the easiest of the three to operate and the hardest to move off of.

When to Actually Pick Each One

• Pick Kafka when you have genuinely high event volume, multiple independent consumer groups need to read the same stream of events, you need to replay or reprocess history, or you're feeding an event-sourced or analytics pipeline where "an immutable log of everything that happened" is the actual point, not a side effect.
• Pick RabbitMQ for task and work-queue distribution, when your routing needs are genuinely complex (fan-out to several services, pattern-based topic routing), your volume doesn't require Kafka-scale throughput, and you want broker-native features like per-message priority, delay, or TTL without bolting on extra infrastructure.
• Pick SQS when you're already on AWS, want zero infrastructure to operate, and your ordering and throughput needs genuinely fit inside Standard's best-effort model or FIFO's real, documented limits — it's an especially good fit as plumbing between Lambda functions and other AWS services without running a broker of any kind.

Frequently Asked Questions