System Outage
Incident Report for Apex Networks Ltd
Resolved
All systems are now operational.
Posted Jul 11, 2022 - 01:17 BST
Update
One database server appears to have become corrupted during the outage and is no longer functional. It is in the process of being restored from backups.
Posted Jul 10, 2022 - 23:13 BST
Update
There are a small minority of RMS users that we are still triaging/resolving issues and hope to have online ASAP.
Posted Jul 10, 2022 - 21:48 BST
Monitoring
We have resolved the root cause of the thermal event and are starting to see recovery for impaired EC2 instances within the EU-WEST-2 Region. At this stage, the vast majority of EC2 instances have recovered and we continue to work on the instances that are still affected.
Customers should now be seeing recovery for instance impairments, and we expect to see recovery for the vast majority of instances within the next hour. We will continue to provide updates as recovery continues.
Posted Jul 10, 2022 - 20:30 BST
Update
We continue to investigate instance impairments to a single Availability Zone in the EU-WEST-2 Region. We have experienced an increase in temperatures within a single data center in the affected Availability Zone, which in some cases has caused impairments for instances in the Availability Zone. We have engineers within the affected data center and are working to resolve the issue.
At this stage, we do not have an ETA but still expect it to be more than an hour.
Posted Jul 10, 2022 - 20:25 BST
Update
The issue is a thermal event in one of Amazon's datacentres, and unfortunately the scale of the error is preventing us from moving resources out. No ETA for a full restore is provided, but they have advised it will be at least another 30 minutes.
Posted Jul 10, 2022 - 19:40 BST
Identified
A number of AWS resources are currently experiencing an outage which has affected service. We are working to get around the issue and bring everything back online.
Posted Jul 10, 2022 - 19:13 BST
Investigating
We are currently investigating a total loss of service for some RMS systems
Posted Jul 10, 2022 - 18:47 BST
This incident affected: Apex RMS.