Major outage with Channel Manager

Incident Report for BigCommerce

Resolved

This incident has been resolved.

Posted Nov 12, 2017 - 00:14 CST

Monitoring

All service have returned to normal operation. We sincerely apologize for the interruptions.

Posted Nov 12, 2017 - 00:06 CST

Update

At this point, all other services have returned to normal. However, we are still working to bring Channel Manager back online.

Posted Nov 11, 2017 - 23:12 CST

Update

Webhooks are functional again. We are working to restore Channel Manager.

Posted Nov 11, 2017 - 21:42 CST

Update

The cluster lasted longer this time but all services are affected again. We are restarting all services again.

Posted Nov 11, 2017 - 20:57 CST

Update

Most services are back online and processing. Some slow responses may occur. We are working to restore normal performance and functionality.

Posted Nov 11, 2017 - 20:27 CST

Update

Webhooks are now functional, though responding slightly slower than normal. We are working to restore other services.

Posted Nov 11, 2017 - 20:06 CST

Update

There is no change in status. All engineers are still working to restore the affected services. Our apologies for the interruptions.

Posted Nov 11, 2017 - 18:32 CST

Update

All available engineers are working on this issue. We expect a more positive update soon.

Posted Nov 11, 2017 - 16:47 CST

Update

The solution that was put in place was stable for about 15 minutes and then became unstable. We are bringing on more engineers to find a better solution.

Posted Nov 11, 2017 - 15:40 CST

Update

We have restored part of our messaging infrastructure. A limited amount of services are working again. We are working to restore the rest.

Posted Nov 11, 2017 - 15:24 CST

Update

We are still working to recover the affected services. A new approach is underway and we hope to have a more positive update soon.

Posted Nov 11, 2017 - 14:28 CST

Update

A fix has been deployed but was unstable. We are trying another approach. Also, BigCommerce new trial stores are currently offline.

Posted Nov 11, 2017 - 13:32 CST

Update

BigCommerce engineers continue to work on the cluster that controls messaging for the platform. This has affected Channel Manager, Webhooks, API, Control Panel and Omnichannel Checkout. We'll update as we make progress.

Posted Nov 11, 2017 - 12:52 CST

Update

BigCommerce Engineers are still attempting to fix the affected services.

Posted Nov 11, 2017 - 11:46 CST

Identified

BigCommerce engineers have identified a performance issue with multiple services which would cause delays in Webhooks and Channel Manager as well as some API and Control Panel requests. We are working on a fix for the underlying issue.

Posted Nov 11, 2017 - 10:52 CST

This incident affected: API & Webhooks, Control Panel and Client Services (Client Portal).