A good portion of Alibaba Cloud providers skilled disruptions following a fireplace at a Digital Realty datacenter in Singapore.
The blaze, which occurred early Tuesday morning on September 10, 2024, is believed to have been triggered by a lithium-ion battery explosion within the facility’s battery room.
Native studies confirmed that Singapore’s Civil Defence Pressure responded to the hearth shortly after it broke out.
In accordance with a spokesperson for Digital Realty, the hearth alarm on the SIN11 datacenter was triggered at 07:45 native time.
All on-site personnel have been safely evacuated by 08:15, and fortuitously, no accidents have been reported.
Alibaba Cloud, one of many largest cloud service suppliers within the area, instantly started detecting anomalies in its Singapore-based Availability Zone C round 10:20 SGT.
The Chinese language cloud big launched a press release confirming the explosion and subsequent fireplace, which led to abnormally excessive temperatures within the affected areas. This, in flip, brought on some cloud providers to expertise disruptions.
Regardless of the hearth, Alibaba Cloud reported that its catastrophe restoration methods have been activated, and failover routines labored as designed.
This meant that its high-availability cloud merchandise continued to function inside promised service ranges. Nevertheless, some customers have been nonetheless required to manually migrate workloads away from the affected zone to make sure continuity.
In accordance with the cloud supplier’s standing updates, sure providers and choices have been briefly taken offline as temperatures throughout the datacenter remained dangerously elevated.
By 20:04 SGT on Tuesday, the state of affairs had not but been absolutely resolved, with workers unable to re-enter the blaze-hit constructing.
By early Wednesday morning (01:46 SGT), additional problems arose because the datacenter started to expertise water leaks, doubtless brought on by firefighting efforts.
This posed a brand new threat {of electrical} quick circuits, forcing an emergency shutdown {of electrical} methods in one of many buildings.
Digital Realty confirmed that at 01:45 SGT on Wednesday, parts of the power had been powered down to stop additional harm. Alibaba Cloud, in the meantime, labored to revive community providers to unaffected buildings within the advanced.
The incident not solely disrupted Alibaba Cloud’s operations but in addition brought on service outages and degradation for different platforms depending on the identical infrastructure. Digital Ocean, IaaS service Coolify, and content material supply community Cloudflare have been all affected by the blaze, in line with on-line complaints from customers.
This incident provides to the rising listing of considerations over lithium-ion battery security in vital infrastructure like datacenters. French cloud supplier OVH misplaced considered one of its datacenters in 2021 resulting from an analogous fireplace in an uninterruptible energy provide (UPS) system.
