9/2/12 Denial of Service Attack Updates: Resolved and All Services Restored
- Jason@DPD
- September 2, 2013
- 39 Comments
Our server is currently under a denial of service attack. This means that an attacker is flooding some of our IP addresses with garbage requests and preventing us from serving cart pages for legitimate requests.
The primary IPs affected include the *.dpdcart.com domain and service to the admin panel is not currently affected. However, because the dpdcart.com domain is affected this does affect adding products to the cart and checkout.
We are working with our datacenter to mitigate the attack, including installing network appliances designed to stop these types of attacks, and we hope to have service restored for everyone shortly.
We will be posting updates to Twitter and here as they happen.
10:40AM Update: Services are being restored now. Change should propagate over all affected IP addresses over the next few minutes.
12:30PM Update: A Cisco threat mitigation appliance has been installed and networks where the attack are originating from are being blocked. Some users who share these networks may be blocked also. This is unavoidable but blocking these networks allows the vast majority to get through.
4:30PM Update: Network traffic is returning to normal, but there are still a few networks and ISP that are being blocked due to the ongoing attack. We’ll continue to monitor and report our status, and a large majority of traffic is currently getting through.
7:30PM Update: Datacenter is reporting all traffic is back to normal and DDoS event is over (hopefully). Everyone should be able to access their cart and checkout.
7:32PM Update: DDoS’d again, twice as hard as before. We’re working on it.
10:30PM (EST) Update: Once again, traffic is starting to come through to the DPD Server. We’ve placed the server behind a threat management appliance so some networks where the distributed attack are originating will continue to be blocked, but for many locations carts will be accessible again.
We’re working with a 3rd party cloud provider to set up continuous DDoS protection by blocking the traffic before it reaches our server. We’ll be up all night getting this set up so we can get this behind us and return to full service for everyone.
We know this has been hard on everyone and we’re taking every step possible to restore service to everyone as soon as possible.
10:30AM Update: DPD is still being protected by the DDoS mitigation appliance and is serving requests to most traffic. The mitigation appliance is still blocking some networks where the ongoing attack is originating from, but most requests are coming through.
There was another attack spike this morning which was mitigated.
We’re still working with the cloud based DDoS protection provider to get our servers protected so we can get off the appliance, but its a very slow and complicated process. We’re working with their tech team to get it in action as soon as possible.
We’re working in rotating shifts to keep people on the job 24/7 during the attack.
6PM Update: New threat mitigation system in place, services are returning to normal for the cart domain. For full details please check your email or read online here: eepurl.com/EFUlX
by David
Sep 02nd, 2013
FYI – my cart pages are still not working (at 12:38pm EST).
Thanks for your attention to this
-David
by Jason@DPD
Sep 03rd, 2013
I’m replying to your comment in addition to updating the post in case you have reply notifications enabled.
New DDoS mitigation systems in place, services are returning to normal. http://eepurl.com/EFUlX
by Jason@DPD
Sep 02nd, 2013
David,
I just tested your cart and it is up and responding. More than likely your browser or computer is caching DNS.
To clear your DNS on windows do the following:
1. Start > Run and enter “cmd” to open the command line.
2. type “ipconfig /flushdns” in to the command line window without the quotes
3. Close your browser and then re-open it and try your cart URL.
On a mac, to flush DNS:
1. Open terminal
2. Enter: sudo dscacheutil -flushcache
3. Enter your password to confirm
4. Close your browser and then re-open it and try your cart URL.
by Michael
Sep 02nd, 2013
My cart isn’t working either. I flushed DNS and tested in all major browsers, but the cart simply times out with a “This webpage is not available” error.
I tried it on two separate network connections with the same result.
Not really sure what timezone you are in – I’m in CEST but the time in UTC is 19:22 now.
by Jason@DPD
Sep 03rd, 2013
I’m replying to your comment in addition to updating the post in case you have reply notifications enabled.
New DDoS mitigation systems in place, services are returning to normal. http://eepurl.com/EFUlX
by Paddy Moogan
Sep 02nd, 2013
Hi Jason,
Thanks for working on this. Currently, my cart URLs are not working either. Tested on a few fresh laptops / browsers and have the same result.
Cheers.
Paddy
by Jason@DPD
Sep 03rd, 2013
I’m replying to your comment in addition to updating the post in case you have reply notifications enabled.
New DDoS mitigation systems in place, services are returning to normal. http://eepurl.com/EFUlX
by Jason@DPD
Sep 02nd, 2013
Paddy and Michael, I just posted an update above. The datacenter reports all traffic is normal now, so if you’re still having trouble please submit a ticket and I’ll work with you one-on-one to figure out what is going on.
by Linda Cruz
Sep 02nd, 2013
Ugh guys your killing me!!! May I suggest when services are DOWN due to any kind of issue that you guys are aware of you NOTIFY customers via email so we can make alternative arrangements??? I have a labor day sale on my webinar and I don’t find out until later that the cart isn’t working… then you tell me that its fixed and now I find out it isn’t working again…. several customers emailed me to request an alternative way to pay… how many DIDN’T?? 🙁
by Dave
Sep 03rd, 2013
I have to agree with Linda here. Your service has been awesome thus far, but I too am in the midst of a big labour day / “back to school” sale and have been receiving a lot of email “feedback” from my potential customers.
Let’s just hope they return and try again.
by Jay
Sep 02nd, 2013
I agree with Linda, this is VERY bad news (lost money)… we just had someone write us saying they were trying to purchase our product but couldn’t because the cart was down.
By the way, I had ZERO idea this was an issue until I got this note from the customer… wouldn’t you think an email is a good idea?
I’m not trying to figure out a viable alternative to DPD in case this happens again.
by Jason@DPD
Sep 02nd, 2013
Jay and Linda,
I’m sorry about no email, but we’ve been posting constant updates here, on Twitter, and we put a notice up in the DPD admin for every vendor. The vast majority of vendors are opted out of our newsletter so we can’t legally email them anyway and twitter/blog/notice in the admin is much more effective at notifying everyone.
For most of the day traffic was getting through normally- we’ve had a period this morning and now this evening where traffic is getting blocked due to the attack.
We’re making every effort to mitigate the attack, but these distributed attacks are nearly 100% out of our control.
by Jay
Sep 02nd, 2013
Jason-
I get that these attacks are out of your control, certainly wasn’t blaming you “for” the attack. I’m a little surprised that there wasn’t some kind of “plan B” (back-up) plan in the event that this happens. DPD is a solid service that many have put their trust in, I would have hoped that a DDoS attack would have been considered as a possibility ahead of time.
by Jason@DPD
Sep 02nd, 2013
Jay,
As one of the owners of DPD I appreciate the trust you place in us. I want you to know that we did have a plan in place that included Cisco Guard at our host, but this attack overwhelmed that system. We’re now working on getting a cloud CDN based system set up for ongoing protection.
by Linda Cruz
Sep 02nd, 2013
I completely understand its out of your control and appreciate your hard work in getting everything resolved. I’m not jumping ship or anything, things happen. I’ve added a backup paypal invoice link to my listing until this situation is completely resolved, it just would have been nice to be able to do this 11 hours ago when the problem started.
by Kloe
Sep 02nd, 2013
Hi there,
Any idea (ballpark) on how long it may take before restoring?
Thanks!
by Jason@DPD
Sep 02nd, 2013
Kloe-
The basic flow for how these things work is this-
1. Someone starts a DDoS attack through a botnet of compromised computers and servers, flooding the target server (DPD) with data. In our case, it was between 2.5-4gbps/sec, which is a lot.
2. The datacenter, to mitigate the attack and protect their network, disconnects us from their network and nullroutes the traffic. We have absolutely no control over this and it’s standard procedure
3. Once the traffic gets to a level they can manage, our server is placed on a threat management appliance and partial service is restored. The networks / ISPs where the threat is still coming from are still blocked, but “clean” traffic is allowed through
4. Once the attack has stopped, they take the threat management appliance off and allow all network traffic like normal.
We’ve been through this entire series already once today, and now we’re back to #3 above on our second round.
We’re working to add a 3rd party service that will provide continuous DDoS protection so we don’t go through it again. We’re going to be burning the midnight oil all night working on it and all hands are on deck to get this resolved.
by Kloe
Sep 02nd, 2013
Hi Jason,
Thanks for the prompt and thorough response. Glad to know a 3rd party alternative is being set-up! It now seems to be working on our end.
Thanks again.
by Brian Lucas
Sep 02nd, 2013
What time zone are these updates? It’s 9:13PM Central Time and my cart isn’t loading either.
by Jason@DPD
Sep 02nd, 2013
We’re Eastern.
by Brian Lucas
Sep 02nd, 2013
I’m back up and running! Thanks Jason and the team for the hard work and the updates! Wish there was a way to sue those involved in the attack.
by Jason@DPD
Sep 02nd, 2013
Kloe-
The basic flow for how these things work is this-
1. Someone starts a DDoS attack through a botnet of compromised computers and servers, flooding the target server (DPD) with data. In our case, it was between 2.5-4gbps/sec, which is a lot.
2. The datacenter, to mitigate the attack and protect their network, disconnects us from their network and nullroutes the traffic. We have absolutely no control over this and it’s standard procedure
3. Once the traffic gets to a level they can manage, our server is placed on a threat management appliance and partial service is restored. The networks / ISPs where the threat is still coming from are still blocked, but “clean” traffic is allowed through
4. Once the attack has stopped, they take the threat management appliance off and allow all network traffic like normal.
We’ve been through this entire series already once today, and now we’re back to #3 above on our second round.
We’re working to add a 3rd party service that will provide continuous DDoS protection so we don’t go through it again. We’re going to be burning the midnight oil all night working on it and all hands are on deck to get this resolved.
by Greg Thurman
Sep 02nd, 2013
Lost sales are a bummer for sure, but I stand by DPD service and support. Good luck with getting this attack under control, and I’m sure this process will make DPD a stronger company going forward. Thanks Jason and crew for burning the midnight oil, as I’ve come to expect nothing less from these guys!
by Dave
Sep 03rd, 2013
Thanks Jason for all the work you guys are doing to protect us. We have a big advertising campaign going out on Wednesday so we are hoping this is fixed tomorrow in time for some good traffic to our store. Keep up the great work!
by Tom
Sep 03rd, 2013
Cart is still down for me. Any updates?
by Jason@DPD
Sep 03rd, 2013
I’m replying to your comment in addition to updating the post in case you have reply notifications enabled.
New DDoS mitigation systems in place, services are returning to normal. http://eepurl.com/EFUlX
by Masa
Sep 03rd, 2013
Same here, cart is down.
by Jason@DPD
Sep 03rd, 2013
I’m replying to your comment in addition to updating the post in case you have reply notifications enabled.
New DDoS mitigation systems in place, services are returning to normal. http://eepurl.com/EFUlX
by Sam
Sep 03rd, 2013
I’m sorry for your trouble, Jason, but I would definitely have appreciated an email to alert me. Not only is it a matter of lost sales, we spend a lot of money on pay-per-click advertising.
As part of your terms of service, you can surely have a clause saying that you will be sending broadcast messages to your customers during an emergency situation like this.
by Kloe
Sep 03rd, 2013
Hi,
While our cart seems to be working, I was wondering what the status was:
We are planning an important launch, and want to know how likely it is that another attack occurs and whether other measures have been successfully put in place to prevent the cart from going down.
Thank you!
by Scott@DPD
Sep 03rd, 2013
DPD is still under attack, but the attack is being mitigated by the threat management appliance that our data center has provided. This allows most clean traffic to DPD through, but some networks from where the threat is coming from are still being blocked.
We are currently working hard to put a permanent solution into place and expect to have that online today.
by Michael Neill
Sep 03rd, 2013
I understand that these things happen–and I’ve been very pleased with the service you offer and have used it for nearly three years to build my business. Like others, a heads up via email would have really been appreciated and maybe DPD customers need to be given an option to get a newsletter solely for the purpose of updates of this type. I was running Labor Day sales and didn’t know there was a problem until I received emails from customers who were unable to process orders.
by Sam
Sep 03rd, 2013
Jason & the rest of the DPD Crew,
We really appreciated your update email today … especially the way you took responsibility in the “What We Could Have Done Better…” paragraphs.
Hope you are all getting on top of the situation now and will be able to catch up on some well-deserved rest.
by Michael P.
Sep 07th, 2013
Can you check my service, if you look at my daily sales they dropped to nothing on Sep 2nd and have been only limping along since. One of the purchases was me testing it, I think I have only made $100 in sales since Sep 2nd and I average more than 250 a day normally. I hope you can get this fixed because my business is hurting pretty bad.
by Jason@DPD
Sep 09th, 2013
Leslie, The DPD servers were never compromised and there is absolutely no chance that a virus was inserted in your file by DPD. DDOS attacks work by “blocking the doorway” for people to connect to your site (think of it as 100 blocking the doorway to a retail store where real customers can’t get in to buy things) and not an attack on our physical servers themselves.
I have serious doubts that a customer even downloaded a virus from a file in DPD because DPD does regular, daily virus scans of all files on our servers, including product files. If we find an infected product that a vendor has uploaded we quarantine it and advise the vendor.
In the case that a brand new threat that our virus scanner doesn’t detect got through to your customer, it was because it was uploaded with the virus already in it- there is no way for the virus to infect the file once its stored in our secure environment.
More than likely your customer had a virus or malware problem around the same time that they downloaded your files, or malware on their computer somehow infected their download when it got to their computer and they just assumed it came from you.
by Michael P.
Sep 10th, 2013
Well I don’t know about anyone else but I am still not getting any sales and there was no response to my last message left on the 7th. I would like to know if the attacks are still going on, I am paying CPC Advertising through the nose here and I need to know if I should pause it. Please respond.
by Jason@DPD
Sep 10th, 2013
Michael,
The attack was over on September 2nd. Any change in sales after the 2nd is not the result of the DOS attack.
I’ll be happy to look at your sales if you submit a support ticket, but I can not talk about your account in a public blog comment.
by Michael P.
Sep 10th, 2013
Okay great, thanks for responding. I only mention my sales because if you look at the graph it is a pretty significant change after the 2nd. I will take your word for it that they are unrelated. Thanks for getting back to me and I agree with everyone else that the service here is really good.
by leslie jackson
Sep 10th, 2013
Thanks for your reply, Jason. I heard from the customer today who said the virus was not related to the download of the book he bought here.
Good luck!