Network connectivity between GCP and our legacy data centers interrupted
Incident Report for BigCommerce
Resolved
All issues have been resolved. This incident was caused by multiple encrypted tunnels between GCP and our legacy hosting provider failing due to issues with GCP's Cloud Routers. The migration of all services to GCP will be completed soon. We have added additional encrypted tunnels to other GCP regions to help insulate against a recurrence of this issue.
Posted Apr 04, 2019 - 19:27 CDT
Monitoring
All services have returned to normal performance metrics. Engineers will continue to monitor for any lingering issues.
Posted Apr 04, 2019 - 19:15 CDT
Update
Engineers continue to work on some services that are performing below target metrics. The performance graphs on this page will continue to be blank until monitoring is restored.
Posted Apr 04, 2019 - 19:09 CDT
Update
Network connectivity has been restored. Engineers are working to resolve lingering performance issues if they exist.
Posted Apr 04, 2019 - 18:54 CDT
Update
Engineers have discovered an issue with the underlying API infrastructure that is affecting many parts of the BC Platform related to this issue. We are working to resolve that issue first.
Posted Apr 04, 2019 - 18:49 CDT
Update
We apologize for some confusion. There were two incidents opened at the same time. We have deleted the second incident where the information on affected components was incorrect. Also, please ignore the performance graphs on this page for now. Due to a configuration error, performance metrics were disabled earlier today. We will be restoring them soon.
Posted Apr 04, 2019 - 18:32 CDT
Update
We have determined that the issue is with GCP Cloud Routers. BigCommerce engineers are attempting to route around this issue to restore the services remaining. Most of the stores using Storefront service have been restored.
Posted Apr 04, 2019 - 18:24 CDT
Update
We are continuing to work on a fix for this issue.
Posted Apr 04, 2019 - 18:22 CDT
Update
Manage.bigcommerce.com has been returned to normal operations. Engineers at our providers are working on the remaining issues.
Posted Apr 04, 2019 - 18:03 CDT
Identified
Our providers have identified an issue and are preparing a fix.
Posted Apr 04, 2019 - 18:01 CDT
Update
Storefront service is also partially down right now for the small number of stores converted to this service.
Posted Apr 04, 2019 - 17:54 CDT
Investigating
Our engineers are investigating connectivity issues between our legacy data center and our current provider. This will mainly affect the few stores not already migrated plus any static asset (images, css and Javascript) which weren't in the CDN at the time. Access to manage.bigcommerce.com is also interrupted. We are working with the providers to fix.
Posted Apr 04, 2019 - 17:52 CDT
This incident affected: API & Webhooks, Control Panel, Storefront and Client Services (Client Portal).