The outbox pattern is boring, and I should have used it two years earlier

#How I got here

It started with Postgres. Then free text search over asset descriptions, so Elasticsearch, written at the end of the same handler. Then similarity search, so vectors. Then a graph, because "show me everything connected to this asset" is a traversal and not a join anyone wants to write in SQL.

A single create request now touched four stores through one handler with a transaction around exactly one of them. I knew this was wrong in the way you know a smoke alarm needs a battery.

#The failure I actually got

I expected a loud failure: search unavailable, request fails, somebody is paged. That happens and it is fine, because you notice.

What happened instead was a deploy that restarted processes mid-request. Around forty writes committed and never reached the search index. Nothing errored, because the process was gone before an error could be logged. Search results were wrong for eleven days, until a customer mentioned that something they had definitely created was not appearing.

The problem was never that a write failed. It was that a write failed and nothing in the system was capable of noticing.

The fix at the time was a reconciliation job. It worked, and it became a permanent fixture nobody owned, running nightly, occasionally repairing in the wrong direction, producing a drift dashboard everyone stopped reading by the third week because the number was always small and never zero.

#What the outbox is

Deliberately unclever. A command runs in one transaction that applies the state change, appends a versioned event, and inserts an outbox row. Nothing leaves the process during that transaction. Either all three rows are there or none are.

A separate relay, one leader at a time, woken by a database notification rather than polling on a timer, reads the outbox in order and publishes. Consumers apply to each projection, and a consumer that dies has its pending entries reclaimed by another.

1// The whole write surface. There is no second way in.
2res, err := f.Exec(tenantCtx, command.Command{
3    Entity: "asset", Op: command.OpCreate,
4    Payload: &domain.Asset{Name: "Pump 7", SiteID: siteID},
5})
Copy

The property that matters is not exactly-once delivery, which nobody has [1]. It is that the projections are derived. Nothing writes to them except an applier reading the log, so both can be dropped and replayed. There is a test that does exactly that, because a rebuild you have never run is a rebuild that does not work.

#Why not a distributed transaction

Two-phase commit across a relational store, a search index and a graph engine is possible in principle and the coordinator becomes a availability bottleneck: a participant that fails after prepare blocks the others until it recovers [4]. The outbox accepts eventual consistency in the projections and keeps the source of truth strongly consistent, which is the better trade for this shape of system [2] [3].

#What it costs

Read after write is no longer free. The commit is done and the projection is not. The window is small, single-digit milliseconds when the relay is warm, and small is not a guarantee you can put in an API contract [5]. Reads go through typed ports, and the relational port reads the source of truth, so a screen that needs what it just wrote reads it there.

Ordering is per aggregate. Not global. Providing a global order would mean serialising every write.

You operate a relay. Leader election, a lag metric, and someone watching it. That is more moving parts than four writes in a handler, and when the relay stops, lag climbs and alerts fire. When a handler silently skips a write, nothing happens for eleven days.

#What I got wrong first

The first relay polled every 200ms, putting a floor of 200ms on projection visibility for no reason other than implementation. A notification-driven wake, with a slow poll retained as a backstop for lost notifications, took the common case to about 4ms. The poll is still there, runs every few seconds, and almost never finds anything.