Multi-Region Data

LinkedIn has data centers in three US locations:

• Ashburn, Virginia
• Richardson, Texas
• Hillsboro, Oregon

This topology keeps the app near users and allows another region to be used as a backup when one fails. If Virginia is unavailable, traffic can move to Texas or Oregon.

Loading map...

Using multiple regions introduces some complexity:

• Which region should each user be routed to?
• What happens to a session when the region is failed out?
• How do we avoid stale reads, split brain, and lost writes?

Topology

Single-Region

Every application is just a view on some data. You can imagine a simplified version of LinkedIn that is composed of only an application server and a database.

In this model, the application is stateless and talks to one database. Since the users and subscriptions tables are in the same database, you can simply join them in a single query:

SELECT u.id, u.name, s.plan
FROM users u
JOIN subscriptions s ON s.user_id = u.id
WHERE u.id = $1
  AND s.status = 'ACTIVE'

Some downsides of the single-region approach include:

1. Single-points of failure across the stack (one issue can take the entire application down)
2. Distant users have high latency
3. There is nothing to failover to
4. Maintenance is harder to hide
5. The entire application relies on one region's capacity; scaling limits are easier to reach

Highly-Available Database

In practice, a "single-region" database is often not a single machine. A common setup has one primary database that accepts writes and multiple read replicas that copy changes from the primary. If the primary fails, one replica can be promoted and the application can reconnect to the new primary.

Multi-Region

Next, we introduce two additional regions to match the architecture of linkedin.com in the United States:

The first thing that you should ask yourself is "what decides where the user is routed to?". The answer is DNS, but we will take a deeper look at that in the Routing Users section.

By adding multiple zones, we've gained lower latency, regional fault tolerance, and more room to spread traffic. But we have also made the system harder to reason about. Now we need rules for routing users, assigning data ownership, and deciding what guarantees replication provides during failures.

Routing Users

Once upon a time, I had a system design interview at Adyen and I couldn't clearly articulate the mechanism which routes users between regions.

It turns out there are a few ways to answer this, for example Cloudflare and AWS list a few methods which enable their DNS servers to act like load balancers:

1. Simple Routing — always route to the same region
2. Failover Routing — route to a backup region when the primary region is unhealthy
3. Geo Routing — route to the nearest region

I think the answer they were looking for was a primitive like GeoDNS, which would enable a user in Seattle to usually land near Oregon, a user in New York to usually land near Virginia, etc.

GeoDNS works by answering the same hostname with different regional endpoints. The user still asks for the same site, but DNS can return Oregon, Texas, or Virginia depending on where the lookup appears to come from.

GeoDNS Routing

DNS answers the same hostname with the closest regional entry point for the user making the query.

Sticky Sessions

GeoDNS can get a user to a nearby regional entry point, but once the user has an active session, the system may want later requests to keep landing on the same region. For example, if the user issues writes to Texas, that data may not be replicated to Virgina or Oregon yet, so the only region with the "up-to-date" data is Texas, and the user's session should continue in Texas at least until the other regions are updated.

That can be done with cookies, load balancer affinity, or session state stored somewhere shared. The tradeoff is that sticky sessions make the app feel stable during normal traffic, but they also make failover harder because the system has to decide what happens when the region holding that session goes away.

Entity Routing

Entity routing means choosing a home region for each important business object, like a customer or tenant. A request can enter through any healthy region, but work for that entity is routed back to the region that owns it, especially writes. This gives the system one clear place where the current version of the entity lives, instead of letting multiple regions accept conflicting updates and trying to pick a winner later.

A new issue appears during failout: if an impacted region owns some entities, the system needs rules for whether those entities move to a backup region or accept only reads until replication catches up.

Entity Routing

A request can enter through the nearest healthy edge while account writes still route to the region that owns that account.

1. Closest edge
2. Find home
3. Route write
4. Commit v19
5. Replicate

Replication, Consistency, and Failover

Since our application is spread across regions, it has to cope with replication lag and the potential for failures.

Replication

Replication is the process of keeping copies of an entity's state in more than one region. Each write creates a new version, often with an increasing counter, timestamp, or log position. That version gives replicas something concrete to compare: Virginia might have account 123 at version 19 while Oregon is still at version 18. Replication will asynchronously update its peer regions; sending the version 19 update from Virginia to Oregon.

Consistency

Consistency is the promise the application makes about what users are allowed to observe. For this article, the useful guarantees are read-your-writes and monotonic reads: after a user changes their account, later requests in the same session should not fall backward to an older version. Weaker guarantees may be fine for feeds, counters, or cached read paths, but account ownership and session-critical data need a clear rule for where the current version lives.

Replication Lag

Replication lag is the delay between a write being accepted in one region and that write becoming visible in the others. The risky case is not a visibly broken replica, but a healthy-looking replica that is still behind: it can answer requests successfully while returning an older account version, breaking read-your-writes or monotonic reads. In the demo below, Virginia accepts version 19, Texas catches up, Oregon remains on version 18, and failover has to treat Oregon as too stale to serve the account.

Replication Lag

A write may be accepted before every region has copied it, so failover has to know which replicas are current enough to serve.

1. Write v19
2. TX catches up
3. VA fails
4. OR stale
5. Repair OR
6. All v19

Failover

Failover is the controlled process of moving traffic and ownership away from a failed region without changing the application's data contract. A safe failover stops sending traffic to the failed region, chooses a replica that has the committed writes, fences the old writer so it cannot come back and accept conflicting updates, and waits for lagging replicas to repair before treating them as current.

Failure Modes

The three failures to watch for are stale reads, split brain, and lost writes. A stale read sends the user to a region that returns an older version; split brain allows two regions to accept writes for the same entity at the same time; a lost write promotes a region that never received a write the application had already told the user was committed.

At LinkedIn

Traffic Routing and Sticky Sessions

From TrafficShift: Load Testing at Scale:

When a user visits “https://www.linkedin.com” in their browser, requests are sent to one of 13 PoPs distributed globally using GeoDNS. Each PoP runs a service called ATS (Apache Traffic Server). To determine which colo to route a user’s traffic to, ATS includes a custom plugin that verifies the signed cookie present in request headers to identify the data center that the user's request should be routed to and establish a sticky session. If this cookie is not available, the primary data center is not online, or if the cookie has expired, ATS communicates with the stickyrouting service to determine the primary colo the user’s request should be routed to and a cookie is set with data center information for that particular user.

Entity Routing Service

From Entity Routing Service (ERS) load test evolution:

An enterprise business has a concept of a “customer entity,” which requires maintaining strong consistency for all users within the same customer scope. Cross data center replication lag is a problem that leads to members or admins seeing stale, inconsistent data. The impact of this lag is greater for enterprise products because more members are operating on the same data set (high cardinality). To mitigate this, the application does second-level routing using Entity Routing Service (ERS), funneling users from the same contract to a specific data center to achieve consistent data access.

High-Availability MySQL

From Addressing the last mile problem with MySQL high availability

A typical MySQL cluster in our ecosystem consists of at least three components:

• A primary node accepting writes
• One or more replicas for read scaling
• A dedicated replica for backup and ETLs