Transactional Outbox: Side Effects with the Business Write

Direct answer. Writing to the database and then calling an email API, a webhook or an ERP is a dual write: either step can succeed alone. Put an outbox row in the same transaction as the business change. A separate relay reads committed rows and delivers at least once; receivers must be idempotent.

Diagram of the transactional outbox: a request handler writes the business row and an outbox row in one database transaction; after commit, a relay polls or tails the outbox and delivers email, webhook or ERP pushes at least once; receivers apply an idempotency gate. A bottom note warns against committing the business write then calling the network with no durable outbox.
Figure 1. Business write and outbox row in one transaction; the relay handles delivery after commit.

1.0 The dual-write problem

A handler that saves an order and then sends a confirmation email is writing to two systems: the database and the mail provider. The same shape appears when you push a document to an ERP or emit a webhook to a partner.

If the database commits and the network call fails, the business state is true and nobody was notified. If you notify first and the commit fails, the far side acted on something that never existed. A two-phase commit across the database and the mail API is rarely available and usually undesirable. That is the dual-write problem the transactional outbox pattern addresses (see Chris Richardson’s pattern write-up and the AWS Prescriptive Guidance page in the references).

2.0 What goes in one transaction

WriteSame transaction as the business change?
Business row (order, payment, ticket, …)Yes
Outbox row (type, message id, payload, status)Yes
Email / webhook / ERP HTTP callNo — after commit, via the relay
Mark outbox row published / delete itAfter a successful relay attempt (or via log-tailing cleanup)

The outbox row is not a log of what already happened on the wire. It is the durable intent to perform a side effect, recorded only if the business write commits.

3.0 Relay: poll or tail the log

A separate process (or worker in the same service) moves committed outbox rows to the outside world.

ApproachHow it finds workTypical trade-off
Polling publisherSELECT pending rows ordered by id or created_atSimple; works on ordinary SQL; lag and load grow with poll interval and table size
Transaction-log tailingRead inserts from the DB log (WAL, binlog, streams)Lower lag; needs DB-specific tooling; still at-least-once

microservices.io documents both as companions to the outbox. Pick polling until volume or latency forces you to invest in log tailing. Either way, treat publish as at-least-once: a crash between “send succeeded” and “row marked done” will send again.

4.0 Row shape we keep stable

Keep enough columns that operations can answer “what is stuck?” without reading application memory:

  • id — monotonic or ULID; used for ordering and relay checkpoints.
  • message_id / idempotency key — stable for this logical side effect; receivers dedup on it.
  • type — which relay handler runs (email.order_paid, erp.order_push, …).
  • aggregate_key — order id, lead id, device id; for ordering and support queries.
  • payload — versioned JSON the handler needs; avoid dumping secrets.
  • status — pending → publishing (optional lease) → published / failed_terminal.
  • attempts, next_attempt_at, last_error — for backoff and alerts.
  • created_at — and optionally published_at.

Claim a row with an atomic update (pending → publishing with owner and lease expiry) so two relay workers do not send the same mail. On success, mark published (or delete, if you prefer a small table and rely on backups for audit). On retryable failure, increment attempts and set next_attempt_at. After a configured limit, move to failed_terminal and alert a person.

5.0 Ordering and duplicates

For a single aggregate, publish in the order the outbox ids (or sequence numbers) were assigned, so a partner does not see “shipped” before “created”. Across aggregates, parallel workers are fine.

Duplicates are expected. Pair this note with Webhook Idempotency and Retries: the outbox guarantees the side effect is attempted after a successful business write; the receiver’s dedup store guarantees it is applied once. ERP pushes need the same discipline on the far side — a stable business key on the document, as in Integrating Orders with an ERP.

6.0 What not to put on the request path

  • Do not open a DB transaction, call a third-party HTTP API, then commit. Timeouts and locks make failure modes hard to reason about.
  • Do not fire-and-forget a goroutine or queue publish without a durable outbox (or equivalent) if losing that side effect on a crash is unacceptable.
  • Do not treat “HTTP 200 from the mail API” as the system of record for “customer was notified” unless you also persisted that outcome. Prefer: outbox published + provider message id stored on the row.

7.0 Observability

  • Metrics: pending age (p50/p95), publish success/fail rates, terminal failures, relay lag.
  • Alerts: oldest pending/publishing past a threshold; spike in terminal failures after a deploy.
  • Support: look up by aggregate_key and show outbox status next to the business record.

If you are wiring side effects for payments, ERP or device clouds, see Web, Cloud and Device Management, or discuss a project with us.

References

Frequently asked questions

Is an outbox the same as a message queue?

An outbox is a table (or document field) in the same database as the business write, so both commit or roll back together. A queue or broker sits after the relay. The outbox exists so the publish cannot succeed without the business write, and the reverse.

Do we still need idempotency on the receiver?

Receivers still need it. The relay can publish more than once: it may crash after a successful send and before it marks the row done. Consumers must treat delivery as at-least-once, for example with the idempotency key and window approach in our webhook note.

Can we call the email API inside the database transaction?

Avoid it. Holding a DB transaction open while waiting on a network call locks rows longer, and a timeout leaves you unsure whether the mail went out. Write the outbox row, commit, then let the relay do the network work.

Polling the outbox table looks wasteful. Is there another option?

Polling works on any SQL database and is usually enough at modest volume. Where the database supports it, transaction-log tailing (for example reading the WAL or binlog for outbox inserts) reduces lag and load. Both still deliver at least once.

What belongs in the outbox payload?

Enough for the relay to act without re-deriving intent: type, a stable idempotency or message id, the business key, and the fields the downstream call needs. Prefer a versioned schema over an opaque dump of the whole aggregate, so old rows stay readable after code changes.