Partial outage due to CDN provider outage

Incident Report for BigCommerce

Resolved

This incident has been resolved.

Posted Dec 31, 2015 - 18:43 CST

Monitoring

We have added load balancing capacity to help handle all the requests from our CDN provider for JS/CSS/Images that were ejected from their servers and response times and error rates are returning to normal. Our CDN provider has fixed their issues and we are now populating the CDN caching servers at a sustainable rate. We will continue to monitor for any lingering issues.

Posted Dec 31, 2015 - 18:06 CST

Update

We are still trying to fix the issue but it has been identified as being caused by extreme load due to failures with our CDN provider, https://status.fastly.com/. They appear to be close to fixing the incidents. We will update shortly, but are still looking at further ways to mitigate. Unfortunately, we rely on our CDN to handle this traffic so we experienced a temporary overload of our cache servers.

Posted Dec 31, 2015 - 16:30 CST

Update

We are still dealing with intermittent issues in both data centers. Customer impact will be slow or partial page loads, followed by successful page loads. We are working still to isolate the cause and correct.

Posted Dec 31, 2015 - 16:18 CST

Update

We are still attempting to fix. This looks more like a DDOS attack on a particular part of the infrastructure that deals with caching CSS/JS. Storefront templates are still available. Sites will be slow to load and may look incomplete.

Posted Dec 31, 2015 - 16:02 CST

Identified

We have identified an issue with a subset of servers in one datacenter. Engineers are fixing.

Posted Dec 31, 2015 - 15:42 CST

This incident affected: Control Panel and Storefront.