What Happens to an AI Service When One Site Goes Offline

Data centers are built to keep AI applications running for their users around the clock. On the rare occasion that a site goes offline, whether for planned maintenance or because of a power or cooling fault, the business relying on it needs to know whether its AI service can keep going. The answer depends on how the business has set up its computing capacity.

To understand why, it helps to understand what happens during an outage in two different setups: four connected sites that share the capacity, and a single building that holds all of it.

Let’s start with a business whose AI capacity is spread evenly across four connected sites. Each site holds a quarter of the total, so if one goes offline, the business loses 25% of its capacity while the other three, which run the same applications, keep going. That leaves 75% still available to support its AI applications. Engineers call this graceful degradation, where the service shrinks by the share that goes offline and carries on.

The picture changes when all that capacity sits in a single building. A building with backup power and cooling goes offline far less often, but when it does, the business loses access to all of its computing capacity at once.

A store checkout is a helpful way to picture the difference between a network of sites and a single building. A store with four registers can keep serving customers when one register closes, and a line only starts to build if more customers arrive than the remaining three registers can handle. A store with a single register stops serving everyone the moment that register closes, however rarely the closure happens.

A network of connected sites works much like the store with four registers, and a single building works like the store with one:

• A network of sites: when one site goes offline, the business only feels the outage if its demand at that moment is more than the remaining sites can carry.
• A single building: when the building goes offline, the business loses service for as long as the outage lasts, regardless of how much demand it has at the time.

Whether a partial outage reaches the business’s users therefore depends on how much capacity the business needs while the site is down, and the business’s demand rises and falls throughout the day.

The amount of computing capacity a business uses changes with the number of requests its AI applications receive. Much of the time, demand sits well below the full capacity installed, because the business has to install enough capacity for its busiest moments.

The timing of an outage therefore decides whether users notice it. If one of four sites goes offline while the business is using half its capacity, the 75% still running covers that demand comfortably and its users carry on as normal. If the business is using more than 75% at that moment, it falls short until the site comes back online.

The question that matters for a business is how often its demand sits above the capacity left running during an outage. Loss-of-Load Probability is the measure built to answer that question, and the next section explains how it works.

Loss-of-Load Probability, or LOLP, measures how often a business’s demand is higher than the capacity still running. It brings together two things: how often a site goes offline, and how often demand climbs above what the remaining sites can carry at that moment. Two sites being offline at the same time is far less likely than one, so for a four-site network, the question that matters most is how often demand rises above 75% while a site is down.

The store with four registers shows how the two come together. A line only forms when a register is closed at the same moment a rush of customers arrives. Registers close only occasionally and rushes come only at certain times of day, so the two seldom overlap, and customers rarely wait. LOLP measures how often that kind of overlap happens for a network of sites: a site offline at the same time as a spell of high demand.

A network of connected sites may have the occasional outage, and what it changes is what an outage means for the business. A single building that goes offline takes all of its capacity with it, while a network loses only the share held at the site that went offline, and the business’s users notice only if demand at that moment is more than the other sites can carry. LOLP puts a number on how often a partial outage would reach those users, and in doing so, it gives the business a more useful question to ask than whether a site might fail: how often would a site failure actually leave it without enough capacity to meet demand?

The answer to that question shapes how multi-site AI infrastructure is designed. Keeping outages rare still matters, and the bigger aim is making sure that, when a site does go offline, enough capacity remains for the service to keep running. Cerbrec uses LOLP when planning how many sites a business’s capacity should be spread across, so the network is sized around the business’s demand and can carry that demand through an outage.

Readers who want to know more can read Cerbrec’s paper, Sizing Multi-Site Capacity with Loss-of-Load Probability.