Service Outage: Power Loss at Primary Data Center Due to Backup Power Failure
- Jason
- Service Status
This morning the Softlayer datacenter where DPD's primary servers are located experienced power loss. The Softlayer datacenter has both battery and backup generator backup power supplies, but the automatic transfer switch for the backup generators failed, resulting in DPD's servers losing power when the backup battery supplies were exhausted. This resulted in an unclean shutdown of DPD's primary database server. As a result, the database was corrupted on disk. DPD maintains regular backups of database data and we immediately worked to restore the latest backup that happened approximately 39 minutes before the power loss event. DPD is now up and fully operational. We are working to restore the missing data that happened between the time the last backup happened at 6:25 AM EST and the power loss event at 7:04 AM EST. We hope to restore this data within the next 24-48 hours but we'll need to extract the data...