Scheduled Maintenance Event: Database Migration / Upgrade – 26 April, 2017

  • Jason@DPD
  • April 24, 2017
  • No Comments

On Wednesday, 26 April, 2017 at 9PM Eastern (Check time in your time zone) we will be performing a database migration on DPD that will lead to improved performance for all vendors and buyers.

We have run simulations of this move and expect less than 5 minutes of downtime where the site may fail to load when the procedure is started and then degraded performance during the migration, which may last up to 2 hours. During this period of degraded performance certain pages may be slow to load, including “database heavy” pages like purchase and customer list pages, reports pages, and possibly the dashboard depending on your recent sales volume.

We apologize for any slow page loads this process may cause, but it will be temporary. The migration to new database hardware will provide expanded memory and processing power for all database operations and will represent a significant speed increase for all vendors when completed. More importantly, this migration will be to a scalable platform that we can expand on demand as load increases.

Thanks in advance for your understanding during this maintenance event. If you have any questions drop us a line on live chat after logging in, or send us a support ticket and we’ll do our best to help!

Service Outage: Power Loss at Primary Data Center Due to Backup Power Failure

  • Jason@DPD
  • July 20, 2015

This morning the Softlayer datacenter where DPD’s primary servers are located experienced power loss.

The Softlayer datacenter has both battery and backup generator backup power supplies, but the automatic transfer switch for the backup generators failed, resulting in DPD’s servers losing power when the backup battery supplies were exhausted.

This resulted in an unclean shutdown of DPD’s primary database server. As a result, the database was corrupted on disk. DPD maintains regular backups of database data and we immediately worked to restore the latest backup that happened approximately 39 minutes before the power loss event.

DPD is now up and fully operational.

We are working to restore the missing data that happened between the time the last backup happened at 6:25 AM EST and the power loss event at 7:04 AM EST. We hope to restore this data within the next 24-48 hours but we’ll need to extract the data from the corrupt database and insert it in to the new restored database, so it will take some time to complete.

Power Outage Event Timeline:

All times Eastern

  • 6:25 : Last regular database backup.
  • 7:04 : Our datacenter received a power outage. Redundant power failed to take over before battery backups were exhausted. The pod our servers are located in then lost power.
  • 9:40 : Power was restored.
  • 9:41 : We discovered an anomaly with the database. Further investigation shows on-disk corruption from the power outage.
  • 10:10 : It was determined that the database was not recoverable in a reasonable time and we would restore from backup.
  • 10:45 : Recovery complete. Approximately 39 minutes of data was lost in the recovery.
  • Ongoing: We’re working to restore the missing data between the last backup and the outage. Updates will be posted as they happen.

We responded to support requests through our help desk, email, Twitter, and Facebook during the outage. We are looking into better ways of relaying system status to you in the future, including a 3rd party status monitor everyone can check.