At approx 4:15pm Friday the UPS in our Huntingdon data centre totally failed. Rather than switching to bypass power (raw mains) as it should have done it turned off all power fed from it.
Our core network in Huntingdon has a separate UPS which stayed on but many servers did power down.
We switched over to raw (unprotected) mains within 10-15 minutes and a majority of servers had their services restored. We did have one rack with a failed network switch which was replaced approx 9pm and a number of servers which needed manually bringing back to service.
Thanks to our UPS service company a replacement 80KVA UPS was installed during the day yesterday (Saturday) and all load was switched back to this restoring service ans resilience to all servers there.
There are a few final jobs to do but these should not be customer affecting. IF you are still seeing any issues please do let us know
Note this outage should NOT have affected ADSL customers as long as you are using the recommended DNS servers as these are not located in Huntingdon – the only services affected *should* have been those that are there.
If you have any questions at all please contact support
We are sorry for anyone affected by this issue