Status.BigCommerce

Resolved @ 10:54 CST Database connectivity to all stores on server4400.bigcommerce.com has been restored. All stores are now up and operational. BigCommerce System Administrators have determined that the outage was caused by uncorrectable memory errors in the server and will be working with our data center to have emergency maintenance on the server scheduled for later in the week and the bad memory replaced.

Update @ 10:44 CST The file system check on server4400.bigcommerce.com has completed successfully. We are now in the process of restoring database connectivity to all stores.

Update @ 10:33 CST The file system integrity check on server4400.bigcommerce.com is continuing to progress.

Update @ 10:07 CST server4400.bigcommerce.com appears to have automatically rebooted for an unknown reason and has initiated a file system integrity check. All stores will be unavailable while this occurs. We are still investigating the cause of the reboot.

Post @ 10:00 CST BigCommerce System Administrators have been alerted of availability issues on server4400.bigcommerce.com. Investigations are currently underway.

Resolved @ 13:17 CST The crash recovery has completed and all stores on server800.bigcommerce.com should now be available. As such, BigCommerce System Administrators are marking this incident as resolved and will continue to monitor the server over the next 48 hours.

Update @ 12:55 CST The filesystem check has completed and the server has now rebooted. A database crash recovery is now being performed and all stores should now be available momentarily, however certain stores may be slow to load initially as database caches are warmed.

Update @ 12:50 CST BigCommerce System Administrators are in the final stages of completing the offline filesystem check. Another update will be provided shortly.

Update @ 12:08 CST Our Service provider have diagnosed a hardware fault and are in the process of replacing it now.

Update @ 11:33 CST BigCommerce System Administrators have found an issue with the RAID controller on server800.bigcommerce.com. We are currently working with our service provider to resolve the problem.

Update @ 11:13 CST The filesystem has gone into a read only state on server800.bigcommerce.com. BigCommerce System Administrators are currently rebooting this server and performing a check of the filesystem. All stores will be unavailable while this occurs.

Post @ 11:05 CST BigCommerce System Administrators have been alerted of availability issues on server800.bigcommerce.com. Investigations are currently underway.

Resolved @ 21:11 CT - The integrity check of the databases has completed and stores should now be available on server4900.bigcommerce.com. BigCommerce System Administrators are still continuing their investigations.

Update @ 21:02 CT - BigCommerce System Administrators have diagnosed the problem and the database server is currently undergoing an integrity check. This post will be updated with more information once it is available.

Post @ 20:51 CT – BigCommerce System Administrators have been notified of availability issues with email accounts and stores on server4900.bigcommerce.com. We are currently logging in to investigate.

Resolved @ 05:25 CT A reboot of server4900.bigcommerce.com has completed and store performance issues are resolved. BigCommerce System Administrators will continue to monitor the server over the next 24 hours.

Post @ 05:16 CT  BigCommerce System Administrators have observed performance issues on server4900.bigcommerce.com due to a possible hardware issue that may be affecting the ability to access some BigCommerce stores. BigCommerce System Administrators are performing an emergency reboot of the server in an attempt to resolve these issues.

Resovled @ 17:30 CT We have mitigated the attack and all stores and services are available.

Update @ 16:40 CT Due to a DDOS attack several of our servers become unresponsive.  We have taken measures to mitigate the attack and store availability is returning to normal.

Post @ 16:20 CT BigCommerce System Administrators have been alerted to several servers with performance issues. We are currently logging in to investigate.

Update @ 17:00 CT The denial of service attack is still ongoing. BigCommerce System Administrators are currently working to mitigate it.

Update @ 16:00 CT The denial of service attack is still ongoing. BigCommerce System Administrators are currently working to mitigate it.

Post @ 14:30 CT Due to a denial of service attack the imap service on server4300.bigcommerce.com is currently unavailable. BigCommerce System Administrators are currently investigating.

Resolved @ 12:45 CT BigCommerce System Administrators have restored access to server4300.bigcommerce.com.

Update @ 12:40 CT server4300.bigcommerce.com was targeted by a DDoS attack, and we have taken steps to mitigate.

Post @ 12:08 CT BigCommerce System Administrators have been alerted of server4300.bigcommerce.com being unavailable. We are currently logging in to investigate.

Resolved @ 20:50 CT The root cause of the issue has been identified and BigCommerce System Administrators will be working on a mitigation plan to ensure it does not happen again.

Update @ 20:37 CT BigCommerce System Administrators have restored access to stores on server4200.bigcommerce.com. We are currently investigating the root cause of the unavailability.

Post @ 20:30 CT BigCommerce System Administrators have been alerted of server4200.bigcommerce.com being unavailable. We are currently logging in to investigate.

Resolved @ 7:15AM CT BigCommerce System Administrators have performed system checks and everything is back to normal operation.  Monitoring of the RAID array will continue.

Update @ 6:45AM CT BigCommerce System Administrators have successfully rebooted the system and found the RAID array to be in a ‘rebuilding’ state.  All stores are now available and we are running checks across the system.

Update @ 6:35AM CT BigCommerce System Administrators logged on and found the OS had put the file-system into RO mode as a precaution to an underlying issue.  We are currently rebooting and will establish the cause.

Post @ 6:30AM CT BigCommerce System Administrators have been alerted of server1500.bigcommerce.com being unavailable. We are currently logging in to investigate.

Resolved @ 10:35AM CT BigCommerce System Administrators have restored access to server4400.bigcommerce.com. All stores are available at this time.

Update @ 10:32AM CT server4400.bigcommerce.com was targeted by a DDoS attack, and we are taking steps to mitigate.

Post @ 10:30AM CT BigCommerce System Administrators have been alerted of server4400.bigcommerce.com being unavailable. We are currently logging in to investigate.