A configuration error in the system controlling emergency generators caused October’s major Google Cloud outage in London, exposing how weaknesses in backup power arrangements can disrupt digital services used by businesses and public organisations.
The incident on 20 October affected more than 35 Google Cloud services, with disruption extending beyond the data centre where the original power failure occurred.
A detailed investigation by Google Cloud identified a fault in the programmable logic controllers responsible for detecting electricity supply failures and initiating the facility’s emergency generators.
The disruption began when an external underground cable fault interrupted electricity supply to a facility supporting europe-west2-b, one of three availability zones in Google’s London cloud region.
Under normal circumstances, the facility’s backup arrangements should have enabled operations to continue while an alternative power source was activated.
However, maintenance work following a generator test on 14 October had left one of the site’s two primary electricity feeds disconnected for repairs. When the remaining active feed subsequently failed, the automated system did not recognise that the facility had lost its usable mains supply.
A control error prevented generator startup —
The investigation found that the controllers were monitoring voltage on the supply side of a disconnected circuit breaker, rather than checking whether electricity was reaching the facility’s operational equipment.
Although the second electricity feed remained live outside the breaker, the breaker itself was open. The facility therefore had no usable supply from that connection.
Both the primary and standby controllers used the same flawed logic and interpreted the remaining voltage as evidence that electricity was available.
Consequently, neither controller instructed the emergency generators to start.
The facility’s uninterruptible power supply batteries maintained operations temporarily but were designed to provide approximately five minutes of electricity while backup generation started.
Once those batteries were exhausted, servers, storage equipment, and network infrastructure within the affected availability zone lost power.
The main disruption lasted six hours and eight minutes, from approximately 07:00 to 13:08 BST, although some individual services and customer workloads experienced different recovery times.
Recovery placed pressure on other systems —
The effects were not confined to equipment within the failed availability zone.
Automated recovery arrangements redirected some workloads towards the two remaining zones in the London region. The additional demand placed pressure on available computing resources, database services, and internal communications between systems.
Google reported processing delays, temporary application errors, and problems affecting some customers whose workloads were not hosted directly within the failed zone.
The disruption affected services including computing, databases, storage, networking, and analytics, with consequences for organisations relying on Google Cloud directly or through other technology providers.
Banking applications, public services, and commercial platforms were among those affected during the incident.
Engineers restored electricity using an alternative supply, checked the condition of affected storage systems, and gradually returned computing capacity to service.
Google subsequently reconfigured and tested the generator controls so that decisions about starting backup power account for the position of circuit breakers as well as electrical conditions.
The company has also introduced additional emergency procedures and testing arrangements, including reviews of comparable controller configurations at other facilities.
Some remediation work remains subject to scheduled completion dates, including further checks associated with the restoration of normal electricity supply arrangements.
The investigation demonstrates how the reliability of business technology services depends on both physical infrastructure and the procedures governing it. A fault in one control mechanism disabled emergency generation, while the resulting transfer of workloads exposed limitations in the remaining cloud capacity.
For organisations reviewing cloud continuity arrangements, the findings provide a concrete example of why the availability of backup systems and additional hosting zones must be assessed alongside the ability of those arrangements to operate during a major disruption.
