Designing a Global Transactional Service

A customer sends a bet to Europe. The database commits it. The reply starts travelling back, and then the connection disappears.

The customer knows only that no reply arrived. The platform knows that money moved. A retry lands in the American region, which is healthy, fast, and dangerously ignorant of what happened a moment ago.

Did the bet happen?

This is where a multi-region architecture stops being a map and becomes an argument about truth. Drawing two boxes is easy. Giving both boxes permission to answer the same question is the hard part.

My first instinct is the familiar one: put a global load balancer in front, replicate the database as quickly as possible, and let the surviving region take over. Then the architecture asks an irritating follow-up.

“Take over from which exact state?”

That question changes the design. The problem is no longer how to send traffic to two places. It is how to make one irreversible decision while the network is allowed to delay, duplicate, reorder, or lose the messages surrounding it.

This guide builds the system from that question outward. The result is globally active at the edge and in its compute, but deliberately conservative where a wrong answer would cost money or corrupt state.

Two routes hide inside one request

Suppose the public API is:

POST https://api.example.com/v1/rounds/abc123/bets

The request needs two routing decisions.

The first is a network decision: where should this connection enter the service? A nearby healthy edge is usually the right answer. It can terminate TLS, absorb attacks, enforce policy, and carry the request over a provider backbone instead of leaving the entire long-haul route to the public Internet.

The second is a state decision: which place is allowed to change round abc123? Geography alone cannot answer that. A user near Virginia might be acting on a round owned in Frankfurt. The nearest server can receive the command, but it should not silently become the authority for it.

The diagram contains the central rule:

Route connections by network health. Route mutations by data ownership.

The first route optimises travel. The second protects meaning.

This is why load-balancer affinity is useful but insufficient. Affinity may send a returning client to the same endpoint while it remains healthy. It cannot prove that the endpoint still owns the row, account, wallet, order, or game round. Correctness has to survive a lost cookie, a fresh client, and the failure of the previously preferred region.

Let the networking tools keep their proper jobs

Several technologies appear interchangeable when they are reduced to arrows on a slide. They become easier to choose when each is given one question to answer.

Weighted DNS answers, “Which address should a new lookup receive?” Route 53 weighted routing is excellent for a 1 percent canary, a blue-green migration, planned maintenance, or a coarse switch between providers. It is a poor place to encode the current owner of round abc123. DNS leaves the request path after resolution, cached answers can outlive a change, and an existing TCP or WebSocket connection cannot be relocated by changing a record. 1 1. A DNS time to live is a cache instruction, not a global invalidation deadline. Recursive resolvers, operating systems, applications, and already-open connections each have their own view of time.

A proxied global load balancer or anycast accelerator answers, “Which healthy regional entry point should receive this connection now?” AWS Global Accelerator provides static anycast addresses and selects endpoints using location, health, and configured weights. Cloudflare’s proxied Layer 7 load balancing remains in the HTTP path, so it can steer individual requests and react without waiting for a recursive resolver to discard a cached answer.

That makes either class a plausible front door for a transactional API. The choice depends on protocols, provider boundaries, customer IP allow-lists, WAF requirements, cost, and how much Layer 7 policy belongs at the edge. Neither choice appoints a database writer.

A tunnel answers another question: “How can the edge reach an origin without exposing an inbound public address?” A Cloudflare Tunnel creates outbound connections from cloudflared to Cloudflare. That can be an elegant origin path beneath Cloudflare Load Balancing. It still does not decide which region owns a transaction, whether a replica is current enough to promote, or whether the old writer has really stopped.

A CDN answers, “Can this observation be served near the viewer without asking the transaction system again?” This is ideal for assets, immutable objects, public results, and read models that are allowed to lag. It is not automatically a write-failover system. CloudFront origin failover, for example, applies only to GET, HEAD, and OPTIONS, not to a failed POST or PUT.

The categories now stop fighting:

DNS          stable naming, canaries, planned and coarse shifts
edge         live connection health, policy, and regional entry
application transaction ownership and command routing
database     serialization and durable commit
events       asynchronous propagation after commit
CDN          cheap global delivery of observations and objects

Complexity often comes from asking one layer to compensate for a missing guarantee in another. A shorter design gives each layer less to pretend about.

The database forces the honest conversation

Ordinary PostgreSQL streaming replication is asynchronous by default. That is useful because the primary can acknowledge a commit without waiting for a distant standby. It also leaves a failure window: the primary can die after acknowledging a transaction but before the standby receives the corresponding WAL record.

PostgreSQL lets us close that window with synchronous replication. The cost is not an implementation flaw. The PostgreSQL replication documentation states the physical minimum plainly: a synchronous commit must wait at least for the round-trip time between primary and standby.

The temptation is to call this a database choice. It is really a product choice.

If a 90 ms write is acceptable and losing an acknowledged transaction is not, synchronous geographic durability may be correct. If a user interaction needs a 30 ms budget, the regions are 80 ms apart, and the transaction must coordinate between them, the arithmetic has already rejected the design. No brand name can negotiate with distance.

Asynchronous replication chooses lower write latency and a non-zero recovery-point risk. 2 2. Recovery point objective, or RPO, is the amount of recent data the system is allowed to lose during recovery. “Zero” is a much more expensive promise than “up to a few seconds,” and an asynchronous replica cannot honestly guarantee zero. Synchronous replication chooses stronger durability and pays in latency and reduced write availability when the required standby cannot acknowledge. A purpose-built distributed SQL database moves the coordination machinery into the database, but it does not make coordination disappear.

Aurora Global Database illustrates the first model cleanly: one primary region performs writes, secondary regions are read-only, and replication is typically under a second. Global write forwarding can accept a statement through a secondary, but the statement is still executed by the primary. The secondary becomes a convenient doorway, not another independent authority.

Aurora DSQL and Spanner represent the other family. They use synchronous replication and quorum machinery to present a strongly consistent distributed database. This can remove a great deal of application-owned failover work. It is the right category when the same logical data truly needs strongly consistent writes through multiple regional endpoints and the application should not assign owners itself. It also brings a new database model, product limits, cost, and WAN-dependent write behaviour.

The question is not, “Which database sounds most global?”

It is, “Which guarantee is valuable enough to put on every write?”

A middle ground: active regions, one writer per entity

Many transactional domains contain a natural unit of serialization:

round_id
wallet_id
account_id
order_id
auction_id
tenant_id

Instead of making Region A the writer for everything, or making every row writable everywhere, assign each entity one owner at a time.

round 781 → Europe
round 782 → America
round 783 → America
round 784 → Europe

Both regional stacks perform real production work. Most commands can execute beside their authoritative data. Yet two regions do not race to settle the same round.

This is active-active compute with single-writer transactional keys. It is particularly attractive when entities are numerous, mostly independent, and easy to place when created. A game round is a good example. An individual round wants strict order, but round 781 does not have to wait for round 782.

It is less attractive when most transactions touch entities owned by several regions. A money transfer between two wallets in different ownership shards can turn the clean local transaction back into a distributed one. Natural ownership key A key is natural when most correctness-sensitive operations stay inside it. If ordinary commands constantly cross keys, the partition boundary is fighting the domain rather than simplifying it.

Start with an explicit record rather than load-balancer stickiness:

CREATE TABLE rounds (
    round_id       uuid PRIMARY KEY,
    owner_region   text NOT NULL,
    owner_epoch    bigint NOT NULL,
    version        bigint NOT NULL,
    status         text NOT NULL,
    state          jsonb NOT NULL
);

CREATE TABLE processed_commands (
    round_id        uuid NOT NULL,
    idempotency_key text NOT NULL,
    response         jsonb NOT NULL,
    PRIMARY KEY (round_id, idempotency_key)
);

CREATE TABLE round_outbox (
    event_id      uuid PRIMARY KEY,
    round_id      uuid NOT NULL,
    round_version bigint NOT NULL,
    event_type    text NOT NULL,
    payload       jsonb NOT NULL,
    committed_at  timestamptz NOT NULL DEFAULT now(),
    UNIQUE (round_id, round_version)
);

This is a teaching schema, not a paste-ready production migration. The important fields are the ones that make ambiguity visible: owner_region, owner_epoch, version, and the two deduplication keys.

At ingress, the command router performs a small piece of data routing:

owner = ownership.lookup(command.round_id)

if owner.region != LOCAL_REGION:
    forward command to owner.region over the private provider path

execute command with owner.epoch

The owner then handles the command in one local database transaction:

BEGIN
  lock the round row
  reject an unexpected owner region or owner epoch
  return the stored response if the idempotency key already exists
  reject an unexpected version
  apply the state change and increment the version
  record the response under the idempotency key
  insert the versioned outbox event
COMMIT

The transaction does four jobs together: it checks authority, orders concurrent changes, makes retries safe, and records what downstream systems must later learn. If any part fails, all of it rolls back.

The public request should carry an idempotency key and, where the domain benefits, an expected version:

POST /v1/rounds/abc123/bets HTTP/1.1
Host: api.example.com
Idempotency-Key: 01K4N6J7V7Q9M7R2N3Q0W5E8F1
Content-Type: application/json

{
  "expectedVersion": 5720,
  "amount": "10.00",
  "currency": "EUR"
}

If the first response is lost, the client retries with the same key. The service returns the recorded result instead of placing a second bet. If another valid command has already advanced the round to version 5721, the stale command receives a conflict instead of overwriting newer state.

The integration guide must state the retry contract: which status codes are retryable, the timeout, exponential backoff and jitter, how long an idempotency key remains valid, and whether the same key with a different body is rejected. 3 3. Store enough of the original request, or a canonical request hash, to distinguish a genuine retry from accidental reuse of the same key for different work. The key should also be scoped to the customer and operation so one tenant cannot collide with another. “Clients may retry” is not a protocol.

The outbox is where the regions learn from each other

After the commit, the other region still needs a copy. So do read models, search indexes, caches, analytics, and viewers. The obvious code is:

update database
publish event

It is also a small trap. The process can die after the database commit and before the publish. Reversing the order merely reverses the inconsistency.

The transactional outbox inserts the state change and its event in the same database transaction. A separate relay or change-data-capture process publishes committed outbox rows. AWS’s transactional outbox guidance highlights the two consequences that matter here: preserve event order, and make consumers idempotent because delivery can repeat.

The receiving region tracks round_version. Event 5722 arriving twice is harmless. Event 5724 arriving before 5723 is detectable. Neither is silently mistaken for a new authoritative command.

Do not confuse an event-fed read model with a promotable failover copy. The recovery path needs durable state that can meet the declared RPO, through WAL or storage replication, or through a complete replayable log whose recovery time has been measured. The outbox carries business consequences safely; it does not by itself prove that a second database is ready to become authoritative.

This gives a useful asymmetry:

commands     travel towards authority
events       travel away from authority
observations come from local derived state when staleness is acceptable

Replicate the durable truth once, then rebuild disposable views from it. Synchronously mirroring PostgreSQL, Redis, search, analytics, and every cache produces more synchronization problems than resilience. A cache should be cheap enough to delete.

Hot, warm, and cold are promises, not storage temperatures

The same guarantee does not belong on every byte.

Hot state can change the outcome of a transaction happening now: the open or closed status of a round, a balance reservation, the current sequence, the command-deduplication record, and the ownership epoch. It belongs beside the authoritative writer, on the shortest correctness path possible.

Warm state is operationally useful but no longer decides the mutation in front of us: recent results, support history, regional read models, and dashboards. A documented amount of staleness is often acceptable here. The local region can serve these reads while events catch it up. 4 4. A user who has just completed a write may reasonably expect to read it back immediately. Carrying the committed version in a token lets the next read wait for that version, route to the authority, or admit that the local view is behind.

Cold state is immutable or archival: replay files, audit exports, logs, media, and backups. Object storage and a CDN are natural tools. S3 replication is asynchronous; S3 Replication Time Control targets 99.99 percent of new objects within 15 minutes. That is an excellent durability and distribution mechanism for an archive, and an absurd mechanism for deciding whether a bet committed five milliseconds ago.

The words “global data” conceal these different promises. Name them instead.

The table is not a mandate to create three new platforms. A small system may keep hot and warm data in one PostgreSQL cluster and move only archives to object storage. The distinction matters before the number of products does.

Failover means transferring permission

Now return to the original failure. Europe is unreachable. America has a recent replica of round abc123.

“Promote it,” says the impatient part of the design.

“What stops Europe from writing when its network returns?” asks the part that has finally learned to be difficult.

An ownership epoch is a fencing token A monotonically increasing generation attached to ownership. A writer holding an older generation is stale even if it still has a running process and an open database connection. . Before failure, Europe writes under epoch 77. A valid promotion changes ownership to America under epoch 78. Every write checks the epoch. The isolated European process may still believe it is important, but its 77 can no longer authorize a mutation.

The epoch is useful only if the decision that advances it is trustworthy. Two regions alone cannot both lose contact and independently infer that the other is dead. If each promotes itself, the epoch becomes decoration.

There are three honest approaches:

• Use a consensus-backed control plane with a third failure domain or witness to grant ownership.
• Use a managed database that internalizes quorum and fencing.
• Keep cross-region promotion deliberate and manual, accepting a longer recovery time in exchange for a much smaller control system.

The third option is often underrated. If a service can tolerate a 20-minute regional recovery and regional failure is rare, a tested runbook may be safer than an automatic election system the team barely understands. If the recovery objective is under a minute at all hours, automation and a third voting failure domain begin to earn their cost.

A conservative promotion sequence is:

1. Stop new traffic from reaching the suspected owner.
2. Establish whether the failure is regional, database-local, or only an inter-region partition.
3. Measure replica lag and compare the possible loss with the declared RPO.
4. Acquire a new ownership epoch from the trusted coordinator.
5. Promote the target data store and enable writes only for that epoch.
6. Move command routing, then verify writes, reads, and event propagation.
7. Rejoin the old region as a follower after reconciling it. Never let it resume from stale local belief.

During uncertainty, returning 503 Service Unavailable with a Retry-After header for the affected entities is often the correct answer. In a financial system, temporary unavailability is visible and repairable. Two contradictory balances are neither.

Health checks should ask whether the service is allowed to serve

A global edge can route around only the failures it can see. GET /health returning 200 OK because the process has an event loop is too weak for transaction traffic.

Separate liveness from readiness:

/live
  process is running

/ready
  process can accept work
  required local database path is usable
  command router has a sufficiently fresh ownership view
  region is authorised for the entities it claims
  replication or failover state is within policy

This does not mean every distant dependency belongs in one fragile health probe. A read-only catalogue endpoint may remain useful when the round writer is unavailable. Health should be capability-specific enough that the edge does not declare a whole region dead because one optional analytics sink is slow.

The important invariant is that connectivity and authority agree. A region that can receive traffic but is fenced from writing must not advertise its mutation endpoint as ready.

Test the pauses between the arrows

The attractive diagram describes steady state. The system is defined by the moments between those boxes.

Test at least these cases before calling the design resilient:

Do not record only whether failover “worked.” Record detection time, routing time, promotion time, the last durable sequence in each region, rejected stale writes, duplicate command results, and the moment derived views converge. 5 5. Availability failover and corruption recovery are different drills. Replication can faithfully copy a bad write to every region, so point-in-time restore, immutable backups, and replay need evidence of their own. Resilience without timestamps is mostly mood.

When the second region earns its existence

The final design choice can be made as a ladder.

Use one write region with a cross-region replica when the mutation latency to that region is acceptable, the workload is modest, and operational simplicity matters more than using both regions for writes. This is the default I would try to disprove first.

Use single-writer ownership per entity when the domain has a strong partition key, traffic is naturally spread, local writes matter, and the team can operate routing, event propagation, and fenced failover. This is the useful middle ground for rounds, orders, tenants, and other mostly independent aggregates.

Use distributed SQL when strongly consistent writes must enter through multiple regions for the same logical data, ownership cannot be kept inside useful boundaries, and the value of database-managed coordination exceeds the latency, compatibility, and cost trade-offs.

Use no multi-region transaction system at all when backups, multi-zone availability, a warm standby, or a tested rebuild already meet the business recovery objective. A second region is not free reliability. It is another system that can disagree with the first.

A practical map from problems to technologies

The ladder is useful, but it is still abstract. In an architecture meeting nobody asks for “the appropriate layer of indirection.” They ask whether five percent of customers can try the new release, whether a partner can keep the same firewall rules, or whether a bet may be acknowledged and then disappear.

So I would choose the machinery by completing one sentence: “I need to…” The precision of the ending usually reveals the right tool.

“I need to send a small share of new connections to a new deployment.” Use weighted DNS, such as Route 53 weighted records. Imagine api.example.com pointing with weight 95 to the established European stack and weight 5 to its replacement. Watch error rates and business outcomes, then move through 80/20, 50/50, and finally 0/100. The same mechanism is useful for planned maintenance, a gradual provider migration, or a coarse switch between two independent global front doors. 6 6. A weight is a relative steering instruction, not a promise that exactly five of every hundred requests will arrive at one destination. Resolver caching, connection reuse, and uneven client traffic make the observed split approximate.

Do not choose weighted DNS because the requirement says, “A dead region must stop receiving transactional requests within seconds.” Some clients will keep an old answer, and an open connection will not move merely because the record changed. DNS is excellent at changing the map handed to future travellers. It does not walk beside each traveller and redirect them at the first closed bridge.

“I need each new HTTPS request to avoid an unhealthy region, and I need HTTP policy at the edge.” Use a proxied global Layer 7 load balancer, such as Cloudflare Load Balancing. Suppose the Frankfurt API fails its mutation-readiness check while Virginia remains healthy. The proxy can send the next request to Virginia, enforce rate limits, reject hostile traffic, terminate TLS, and perhaps route /media/ differently from /v1/bets/. Virginia is now the healthy entrance. It is not automatically the owner of the requested round, so its command router must still forward or reject the mutation according to the ownership map.

“I need stable public addresses, fast regional steering, or a front door for TCP or UDP.” Use an anycast accelerator, such as AWS Global Accelerator. A concrete B2B example is a payment partner that will allow-list only two IP addresses. The accelerator keeps those addresses stable while the regional load balancers behind them change, scale, or fail. A real-time game protocol that is not ordinary HTTP may fit here too. If the design instead needs cookie inspection, path routing, WAF rules, or response caching, a Layer 7 proxy is the more natural instrument. Some systems use both, but they should earn the extra diagnostic surface.

“I need the origin to be reachable without giving it a public inbound address.” Use a secure origin tunnel, such as Cloudflare Tunnel, beneath the load balancer. For example, each region can run two tunnel connectors that establish outbound paths from a private subnet to the edge. The edge receives the public request, chooses a healthy regional pool, and uses the tunnel to reach it. The tunnel removes exposed origin ports. It does not compare database epochs, promote a replica, or decide that Region B may now spend money on behalf of Region A.

“I need many people to receive the same bytes without involving the transaction system.” Use a CDN, such as CloudFront or Cloudflare’s CDN. Put JavaScript bundles, images, replay files, public results, and immutable summaries of completed rounds there. A viewer in Sydney can receive a finalized result from a nearby cache while the authoritative database remains in Europe. Keep open-round state, balances, and POST /bets off that path unless their semantics genuinely permit staleness. A CDN is a distribution machine. It becomes dangerous when convenience quietly promotes a cached observation into a decision.

The same problem-first test works below the network.

“I can afford one distant writer, and a small recovery window is acceptable.” Use one write region with asynchronous cross-region replication, through PostgreSQL streaming replication or a managed one-writer global database. A European business system whose mutations are rare and whose American users can tolerate an extra 80 milliseconds is a good example. The American replica can serve reports and support searches, while all orders still commit in Europe. The design is compact and understandable. Its honest price is that the newest acknowledged transactions may not exist on the replica at the instant Europe disappears.

“I cannot lose an acknowledged transaction, and the inter-region wait fits the product.” Use synchronous cross-region replication. If a settlement service can tolerate a 100 millisecond commit but cannot tolerate an RPO above zero, waiting for a remote durable acknowledgement may be the correct trade. The wait belongs in the user-visible latency budget, and loss of the required standby may stop writes. That is not the system malfunctioning. It is the system keeping the promise that justified synchronous replication.

“I need both regions to perform local writes, but each transaction has a natural home.” Use single-writer ownership per entity. A new game round created for European players can receive owner EU and epoch 41; an American round receives owner US and epoch 12. Both regions are active writers, yet only one may advance a particular round. If a European customer acts on the American round, the European API forwards the command to its owner. When ownership moves, the epoch moves with it, fencing the former writer.

Add a transactional outbox when the sentence continues: “…and other systems must reliably learn what committed.” A successful bet and its BetAccepted event enter the same local transaction. The event relay may publish twice, so the scoreboard deduplicates by event ID and round version. This is the right tool for feeding regional read models, notifications, audit streams, and caches. It is not a substitute for the durable copy from which a failed writer will be recovered.

“The same logical rows must accept strongly consistent writes through multiple regions, and there is no useful ownership boundary.” Evaluate distributed SQL, such as Spanner or Aurora DSQL. Picture a scarce global inventory pool where reservations in Europe and America contend for the same final units, and routing every reservation to one regional owner defeats the latency objective. A database with built-in consensus can be worth its cost here. It moves quorum, leader placement, and fencing out of application code, but every transaction still pays for the coordination its guarantees require. 7 7. The evaluation must include the real transaction shape, not only a single-row benchmark. Cross-row constraints, secondary indexes, schema changes, hotspot behaviour, regional failure, and the slow tail are where an attractive median can become an unsuitable system.

Finally, “I need to recover within four hours, and losing several minutes of recent internal data is acceptable.” Use tested backups, multi-zone availability, and perhaps a warm standby. Do not build a global transaction protocol to satisfy a recovery objective that a restore drill already meets. Complexity is not evidence of seriousness. Sometimes the most mature design is the one that can explain why it stopped adding machinery.

For the platform in this article, the choices now become explicit. Weighted DNS handles canaries and planned provider shifts. A proxied load balancer or anycast accelerator is the everyday global entrance. Tunnels are optional private roads from that entrance to the origins. Ownership routes each command to one regional writer. The outbox carries committed consequences outward. A CDN serves the observations that are safe to copy. Distributed SQL enters the conversation only if transactions routinely cross the ownership boundaries that were meant to keep coordination local.

Each technology is useful. The architectural mistake is promoting usefulness into authority.

That is the uncomfortable conclusion. Multi-region design becomes robust not when both halves can do everything, but when each half knows what it is forbidden to do.

The customer from the opening still sees one stable name. The edge still finds a healthy entrance. Both regions still carry useful work. But the retry in America does not improvise an answer. It follows the command to the owner, or it returns the result already recorded under the same idempotency key, or it waits while a fenced promotion establishes a new owner.

The architecture succeeds by becoming less symmetrical and more truthful.