Dec 31, 2012 | Outages
At approx 3:30pm today there was a power outage from the raw mains to our Huntingdon Data Centre which resulted in the loss of a single phase of the mains feed.
While we have a UPS and Generator backup to the data centre the actual failure coincided with a surge from the data we have here – this lead to our UPS protecting itself and our servers and consequently taking down the power feed to all servers in the data centre.
As soon as we were sure of the cause we manually transferred the power feed to our generator and restored power to all affected routers and servers – most were restored within 20-30 minutes. We are currently going through all remaining servers and checking their status.
We now believe that the main issue has been resolved and all public facing services and co-location servers are back and accessible.
IF you are still seeing an issue please call or email support the normal way and we will investigate and resolve the problem.
We are sorry for any issues that have been caused by this outage
Dec 29, 2012 | Outages
app approx 5am this morning our main outbound email server crashed do to a memory issue.
This has been resolved and all services should have been running again just after 8:30am
We are sorry of this issue affected you in any way
Dec 27, 2012 | Outages
Some of our users may have noticed some periods of packet loss and inability to reach certain locations today.
We have been working on this issue since the early hours of this morning – and after a lot of debugging e have traced this issue to a change made by a supplier to two of our inter-site fibre links.
We are liaising with the supplier to get this fixed and have made some temporary changes to stop the issue re-occurring however this had left the network with less automatic resilience than normal.
There is a small risk of a network break on our other services causing some areas of our network to become unreachable – however if this happens we will manually switch over to the backup links. In addition once or supplier completes work here there will be an at risk period when we swap back to all current links.
Virgin Media have now replaced a faulty NTE in our Huntingdon Data Centre which appears to have now resolved this issue. All of our links and resilient paths are now fully enabled. We will be monitoring this issue t ensure it does not re-occur.
We are sorry for any issues caused by this problem
