The Outbox Pattern Explained: Reliable Events From a Database Transaction

Quick Answer

The transactional outbox pattern fixes a real production bug: writing to your database and publishing an event to a message broker are two separate operations, and if your process crashes between them, you either lose the event or send one for data that was never actually saved. The fix is to write the event into an outbox table inside the exact same database transaction as the business write, then let a separate relay process read that table and actually publish to the broker. We ran this for real against a live database, including a rollback that proves both writes genuinely live or die together.

The Dual-Write Problem This Pattern Actually Solves

The naive version of "save something, then tell the rest of the system about it" looks harmless: a service runs INSERT INTO orders(...), commits, and then separately calls something like kafkaProducer.send(orderCreatedEvent). That's two operations, against two completely different systems, with nothing tying them together — no shared transaction, no atomicity between them at all.

Both failure directions are real, not theoretical. If the process crashes after the database commit but before the publish call, the order exists, permanently, and the event that was supposed to trigger a confirmation email, update a search index, or notify a downstream service simply never fires. If the publish happens first and the surrounding database transaction then rolls back for any reason, you've now broadcast an event for an order that, as far as the database is concerned, doesn't exist. Neither of these is a corner case you might hit; they're both things that will happen, eventually, at real production traffic — a container getting killed mid-request during a routine deploy is an ordinary event, not an edge case.

What makes this bug genuinely dangerous is how invisible it is during development. The window between the database commit and the publish call is usually a few milliseconds, so it can pass weeks of local testing and staging traffic without ever showing itself, then fail just often enough in production — at real concurrency and real deploy frequency — to be almost impossible to reproduce on demand. That profile, reliably rare but not actually rare, is exactly the kind of bug worth designing away structurally instead of hoping never to hit.

The Outbox Table, and Actually Creating It

The fix starts with a second table, living in the same database as your business tables, that records events as an ordinary row instead of an external side effect. Here's the real schema we used, built and run against a live MariaDB server for this post:

CREATE TABLE orders (
  id INT PRIMARY KEY AUTO_INCREMENT,
  customer_name VARCHAR(100) NOT NULL,
  amount_cents INT NOT NULL,
  status VARCHAR(20) NOT NULL
) ENGINE=InnoDB;

CREATE TABLE outbox (
  id INT PRIMARY KEY AUTO_INCREMENT,
  aggregate_type VARCHAR(50) NOT NULL,
  aggregate_id INT NOT NULL,
  event_type VARCHAR(50) NOT NULL,
  payload TEXT NOT NULL,
  created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
  published_at TIMESTAMP NULL
) ENGINE=InnoDB;

aggregate_type and aggregate_id identify what the event is actually about (here, a specific order), event_type is the event name a consumer switches on, and payload holds the serialized event body. The entire mechanism hinges on one column: published_at, nullable, unset until a separate process actually ships the event. One honest note on the schema: this MariaDB instance predates native JSON column support, so payload is stored as TEXT holding a JSON string — functionally identical to what you'd store in Postgres's JSONB or MySQL 5.7+'s native JSON type, just without the engine parsing it for you at the column level.

Writing Both Rows in One Transaction, For Real

Here's the actual transaction, run for real, that writes a business row and its event row together:

START TRANSACTION;
INSERT INTO orders (customer_name, amount_cents, status)
  VALUES ('Aisha', 24900, 'created');
SET @order_id = LAST_INSERT_ID();
INSERT INTO outbox (aggregate_type, aggregate_id, event_type, payload)
  VALUES ('order', @order_id, 'OrderCreated',
    CONCAT('{"orderId":', @order_id, ',"customer":"Aisha","amountCents":24900}'));
COMMIT;

And the real, unedited result immediately after — both rows exist, committed together:

id  customer_name  amount_cents  status
1   Aisha          24900         created

id  aggregate_type  aggregate_id  event_type    payload                                                 published_at
1   order           1             OrderCreated  {"orderId":1,"customer":"Aisha","amountCents":24900}  NULL

The more important test is the failure path. We started a second transaction, inserted a second order and its matching outbox row, queried both back while the transaction was still open (both were visible, exactly as expected), and then issued ROLLBACK instead of COMMIT — simulating the application deciding, for some downstream reason, to abort the whole operation. Querying both tables again afterward showed neither row anywhere:

-- after ROLLBACK --
id  customer_name  amount_cents  status
1   Aisha          24900         created        -- only the earlier, committed order remains

id  aggregate_type  aggregate_id  event_type    payload
1   order           1             OrderCreated  {"orderId":1,...}   -- only the earlier, committed event remains

That's the actual guarantee this pattern rests on: it isn't "we try hard to keep these in sync," it's that the database's own transactional atomicity makes it structurally impossible for one row to exist without the other, because they were never two separate operations pretending to be one — they're one transaction containing two INSERT statements. One honest side detail from the real run: the rolled-back order's auto-increment id was still consumed and never reused (the next successful order jumped from id 1 to id 3). That's a well-known, harmless InnoDB behavior — auto-increment counters are never rolled back, only row data is — worth knowing so a real id gap doesn't get mistaken for a bug later.

The Relay: Turning Outbox Rows Into Real Broker Messages

The outbox table by itself never talks to Kafka, RabbitMQ, or SQS — something else, a separate relay process (sometimes called a message relay or an outbox poller), is responsible for actually publishing. There are two common ways to build one.

The simplest is a polling publisher: a background job that periodically runs a query like the one below, publishes each unpublished row to the real broker, and marks it published only once the broker has confirmed receipt. We ran exactly this sequence for real:

SELECT id, event_type, payload FROM outbox WHERE published_at IS NULL;
-- returns the row: id=1, event_type='OrderCreated', payload={...}

UPDATE outbox SET published_at = NOW() WHERE id = 1;

SELECT id, event_type, published_at FROM outbox;
-- id=1, event_type='OrderCreated', published_at='2026-09-23 22:46:08'

This is simple to build and reason about, but it adds polling latency — typically sub-second to a few seconds depending on the poll interval — and needs its own care to avoid double-publishing if the process crashes between the successful publish and the UPDATE that marks it done.

The more advanced, lower-latency approach is Change Data Capture (CDC), most commonly implemented with a tool like Debezium, which tails the database's own replication or binlog stream — the same low-level mechanism a replica database uses to stay in sync — and turns each new outbox insert directly into a broker message, without ever running a SELECT against the live table. It adds near-zero latency and zero polling load on the primary database, at the cost of operating a CDC pipeline as another moving part. This pattern is often the concrete plumbing underneath a "publish domain events reliably" requirement in the event sourcing, CQRS, and event-driven architectures already covered elsewhere on this blog — it's the mechanism, not a competing idea.

What Guarantee You Actually Get, and the Idempotency You Still Need

The outbox pattern gives you at-least-once delivery, not exactly-once. If the relay publishes a message successfully but crashes before it can run the UPDATE that marks the row published, it will republish that same row after restarting, because from its perspective the row still looks unpublished. This is a deliberate trade-off, not an oversight: the alternative ordering — mark published, then publish — risks the opposite and strictly worse failure, silently losing an event forever if the crash happens between those two steps.

Because of that, consumers reading these events still need to be idempotent, typically by tracking each event's unique id (the outbox row's own id, or a UUID assigned when it was written) and skipping anything already processed. This isn't a limitation specific to the outbox pattern — it's the same idempotency requirement any at-least-once messaging system, including plain Kafka or SQS consumption, already pushes onto whoever's reading the messages.

Running This in Production: Cleanup, Ordering, and When It's Overkill

Outbox rows accumulate, and production setups deal with that in one of two ways: deleting a row once it's confirmed published (relying on the broker and its own durability from that point forward), or periodically archiving old rows rather than letting the table grow forever. Either is fine — the wrong answer is treating the outbox table as permanent storage for the events themselves.

On ordering: a single polling relay reading in id or created_at order publishes in the same order rows were inserted, which is normally all you need, since most consumers only care about ordering within one aggregate — one order's events staying in sequence — not global ordering across every aggregate in the system. That guarantee gets weaker if you shard the outbox across services or run multiple relay instances in parallel without coordination, which is worth knowing before you scale the relay out for throughput.

And it's worth being honest about when this pattern is unnecessary complexity: a single monolith writing to one database, with nothing else needing to react to that write, doesn't need an outbox table at all. The pattern earns its cost specifically at the moment a database commit needs to reliably notify something outside that database — a different service, a search index, a downstream queue — and not a moment before.

Frequently Asked Questions