Transactional outbox, or: how I stopped losing events

You update an order and publish order.paid. Two systems, two writes, no shared transaction. Somewhere between the database commit and the broker publish, the process can die — and now your database says the order is paid and your queue never heard about it. The outbox is how you stop lying about that.

Every team eventually writes one. Here is the version we landed on, and the parts that took a few incidents to get right.

What goes in the outbox — and what doesn't

Not everything. An outbox trades latency and a table for durability; you only want that trade where losing the event would be a correctness bug, not an annoyance. The rule we settled on: if an event must be consistent with a database change, it goes in the outbox — written in the same transaction as that change. If it is fire-and-forget (warm a cache, a non-critical nudge), it dispatches directly.

The signal is literally whether the caller hands us a transaction. One entry point takes an EntityManager and enrolls the event in your transaction; the other just ships:

// MUST be consistent with a DB change -> enroll in the caller's transaction
await queue.createEventWithFallback(manager, { eventType: 'order.paid', ... });

// fire-and-forget -> straight to the broker, no outbox row
await queue.dispatchDirectly({ eventType: 'cache.warm', ... });

Optimistic dispatch, durable fallback

The textbook outbox writes a row and lets a poller ship it "soon." That adds poll latency to every event — a tax you pay even when nothing is wrong. We don't. We write the row inside your transaction, then try to dispatch immediately. If it works — and it almost always does — we mark the row processed inline, so the happy path is exactly as fast as a direct dispatch. If the broker is unreachable, the row simply stays pending, and a background processor takes it from there.

// all inside the caller's DB transaction
await manager.save(outboxEvent);                 // durable: commits with your change
try {
  const job = await dispatchToQueue(event);      // optimistic: ship it now
  await manager.update(OutboxEvent, id, {
    status: 'processed', externalJobId: job.id,
  });
} catch {
  // broker down? leave it 'pending' — the batch processor will retry it
}

Best of both: direct-dispatch speed on a normal day, zero loss on the worst one. The only thing guaranteed to be durable is the intent — the row that committed atomically with your data.

Claiming work without a distributed lock

Several instances run the background processor. How do they avoid grabbing the same rows? Not Redis Redlock, not a consensus library — Postgres FOR UPDATE SKIP LOCKED.

UPDATE outbox_events
SET status = 'processing', processing_instance = $1
WHERE id IN (
  SELECT id FROM outbox_events
  WHERE status IN ('pending', 'failed')
    AND retry_count < max_retries
    AND (next_retry_at IS NULL OR next_retry_at <= NOW())
  ORDER BY priority DESC, created_at ASC
  LIMIT $2
  FOR UPDATE SKIP LOCKED            -- step over rows another worker already holds
)
RETURNING *;

Each instance atomically claims a disjoint batch. SKIP LOCKED means worker B steps over the rows worker A already locked instead of blocking on them — no contention, one round-trip, and the claim is visible right in the row (processing_instance). Ordered by priority then age, so urgent and oldest drain first.

Three heartbeats

The processor runs at three cadences, not one: every 5 minutes to sweep up the handful that failed immediate dispatch, hourly as a steady backstop, and a daily maintenance pass that takes a bigger batch and does the housekeeping. High frequency keeps recovery latency low without a tight poll loop hammering the table.

Dedup, backoff, and the dead letter

A few mechanisms make the at-least-once pipeline safe to lean on:

• Idempotency key — each event can carry a unique key. A client that retries (network blip, double-submit) produces the same key; we find the already-processed event and return its result instead of enqueuing a second job. Exactly-once effect on top of at-least-once delivery — no lock required.
• Exponential backoff — a failed event retries on min(2^n × 30s, 30 min): 30s, 1m, 2m, 4m … capped at 30 minutes, so a flapping downstream doesn't get hammered.
• Dead letter — after max retries the event moves to a dead_letter state with its error and stops retrying. It waits for a human instead of looping forever.
• Self-healing — a row stuck in processing (an instance crashed mid-dispatch) is reset to pending after a timeout and retried. Corrupted rows are deleted so they can't loop.

A cousin worth knowing: advisory locks

SKIP LOCKED hands different rows to different workers. Its cousin solves the opposite problem: when you want exactly one instance to run something — an hourly sweep, a single-leader job — and the rest to stand down. Postgres advisory locks do it with one call:

SELECT pg_try_advisory_lock($1);  -- true for exactly one caller, false for the rest

The instance that gets true runs the job; everyone else skips. The lock is session-scoped, so if the holder crashes it is released automatically and another instance picks up on the next tick — failover with no extra moving parts. Mechanically unrelated to the outbox, but the same instinct runs through both: let the database you already operate be the coordinator, instead of bolting on a second system to keep honest.

The outbox doesn't make publishing reliable — brokers still flake. It makes the record of intent reliable, committed in the same breath as your data, and then never stops trying until intent becomes fact. Slower on the worst day. Correct on every day.