Transactional Outbox: Side Effects with the Business Write
Direct answer. Writing to the database and then calling an email API, a webhook or an ERP is a dual write: either step can succeed alone. Put an outbox row in the same transaction as the business change. A separate relay reads committed rows and delivers at least once; receivers must be idempotent.
1.0 The dual-write problem
A handler that saves an order and then sends a confirmation email is writing to two systems: the database and the mail provider. The same shape appears when you push a document to an ERP or emit a webhook to a partner.
If the database commits and the network call fails, the business state is true and nobody was notified. If you notify first and the commit fails, the far side acted on something that never existed. A two-phase commit across the database and the mail API is rarely available and usually undesirable. That is the dual-write problem the transactional outbox pattern addresses (see Chris Richardson’s pattern write-up and the AWS Prescriptive Guidance page in the references).
2.0 What goes in one transaction
| Write | Same transaction as the business change? |
|---|---|
| Business row (order, payment, ticket, …) | Yes |
| Outbox row (type, message id, payload, status) | Yes |
| Email / webhook / ERP HTTP call | No — after commit, via the relay |
Mark outbox row published / delete it | After a successful relay attempt (or via log-tailing cleanup) |
The outbox row is not a log of what already happened on the wire. It is the durable intent to perform a side effect, recorded only if the business write commits.
3.0 Relay: poll or tail the log
A separate process (or worker in the same service) moves committed outbox rows to the outside world.
| Approach | How it finds work | Typical trade-off |
|---|---|---|
| Polling publisher | SELECT pending rows ordered by id or created_at | Simple; works on ordinary SQL; lag and load grow with poll interval and table size |
| Transaction-log tailing | Read inserts from the DB log (WAL, binlog, streams) | Lower lag; needs DB-specific tooling; still at-least-once |
microservices.io documents both as companions to the outbox. Pick polling until volume or latency forces you to invest in log tailing. Either way, treat publish as at-least-once: a crash between “send succeeded” and “row marked done” will send again.
4.0 Row shape we keep stable
Keep enough columns that operations can answer “what is stuck?” without reading application memory:
- id — monotonic or ULID; used for ordering and relay checkpoints.
- message_id / idempotency key — stable for this logical side effect; receivers dedup on it.
- type — which relay handler runs (
email.order_paid,erp.order_push, …). - aggregate_key — order id, lead id, device id; for ordering and support queries.
- payload — versioned JSON the handler needs; avoid dumping secrets.
- status —
pending→publishing(optional lease) →published/failed_terminal. - attempts, next_attempt_at, last_error — for backoff and alerts.
- created_at — and optionally
published_at.
Claim a row with an atomic update (pending → publishing with owner and lease expiry) so two relay workers do not send the same mail. On success, mark published (or delete, if you prefer a small table and rely on backups for audit). On retryable failure, increment attempts and set next_attempt_at. After a configured limit, move to failed_terminal and alert a person.
5.0 Ordering and duplicates
For a single aggregate, publish in the order the outbox ids (or sequence numbers) were assigned, so a partner does not see “shipped” before “created”. Across aggregates, parallel workers are fine.
Duplicates are expected. Pair this note with Webhook Idempotency and Retries: the outbox guarantees the side effect is attempted after a successful business write; the receiver’s dedup store guarantees it is applied once. ERP pushes need the same discipline on the far side — a stable business key on the document, as in Integrating Orders with an ERP.
6.0 What not to put on the request path
- Do not open a DB transaction, call a third-party HTTP API, then commit. Timeouts and locks make failure modes hard to reason about.
- Do not fire-and-forget a goroutine or queue publish without a durable outbox (or equivalent) if losing that side effect on a crash is unacceptable.
- Do not treat “HTTP 200 from the mail API” as the system of record for “customer was notified” unless you also persisted that outcome. Prefer: outbox published + provider message id stored on the row.
7.0 Observability
- Metrics: pending age (p50/p95), publish success/fail rates, terminal failures, relay lag.
- Alerts: oldest
pending/publishingpast a threshold; spike in terminal failures after a deploy. - Support: look up by
aggregate_keyand show outbox status next to the business record.
If you are wiring side effects for payments, ERP or device clouds, see Web, Cloud and Device Management, or discuss a project with us.
References
- Chris Richardson, Pattern: Transactional outbox: https://microservices.io/patterns/data/transactional-outbox.html
- Chris Richardson, Pattern: Polling publisher: https://microservices.io/patterns/data/polling-publisher.html
- Chris Richardson, Pattern: Transaction log tailing: https://microservices.io/patterns/data/transaction-log-tailing.html
- AWS Prescriptive Guidance, Transactional outbox pattern: https://docs.aws.amazon.com/prescriptive-guidance/latest/cloud-design-patterns/transactional-outbox.html
Frequently asked questions
Is an outbox the same as a message queue?
An outbox is a table (or document field) in the same database as the business write, so both commit or roll back together. A queue or broker sits after the relay. The outbox exists so the publish cannot succeed without the business write, and the reverse.
Do we still need idempotency on the receiver?
Receivers still need it. The relay can publish more than once: it may crash after a successful send and before it marks the row done. Consumers must treat delivery as at-least-once, for example with the idempotency key and window approach in our webhook note.
Can we call the email API inside the database transaction?
Avoid it. Holding a DB transaction open while waiting on a network call locks rows longer, and a timeout leaves you unsure whether the mail went out. Write the outbox row, commit, then let the relay do the network work.
Polling the outbox table looks wasteful. Is there another option?
Polling works on any SQL database and is usually enough at modest volume. Where the database supports it, transaction-log tailing (for example reading the WAL or binlog for outbox inserts) reduces lag and load. Both still deliver at least once.
What belongs in the outbox payload?
Enough for the relay to act without re-deriving intent: type, a stable idempotency or message id, the business key, and the fields the downstream call needs. Prefer a versioned schema over an opaque dump of the whole aggregate, so old rows stay readable after code changes.