Good morning.
We continue to monitor and manage this issue with assistance from our network team and we aim to provide a further update by 11am.
Thanks for your patience and once again, our apologies for the issues seen.
In the early hours of this morning a single PDU failed taking approx 5 servers down in our data centre – including one of our two mail servers (which work as a mirror) and a couple of other customer servers.
The PDU has been replaced and we are just checking that all servers return as they should.
Note that no email etc has been lost here.
We will update this as we have confirmation all servers are back up and operational
[Update 10:30]
All affected services & servers are now back up and running after the PDU replacement – several VMsneeded a manual boot and some services needed a manual start. It’s possible that some older email may take a little while to sync but no email has been lost
We are sorry for the issues here
We are aware of a current issue at the Harbour Exchange Square (HEX) Data Centre
Note that currently none of our equipment at the site is down – but we have seen a number of suppliers we connect to down and a number of other providers with issues too
We have a small number of leased lines down as a result – most auto swapped over to the backup interconnect in Telehouse – there’s 2-3 that we are looking at to manually move.
At this point – we have had a single update from Equinix to say a fire alarm triggered at 4:30 which is when we saw connections drop – we re waiting for a further update.
At this point – other than the small number of leased lines down (which should be back in the next 60 minutes – there are no other issues. However as we don’t have any details of the cause of the issue – we should assume that connections and network there is ‘at risk’
We will update more as we have it
[Update 6:50]
Equinix have just sent the following update
Equinix IBX Site Staff reports that fire alarm was triggered by the failure of output static switch from Galaxy UPS system supporting levels 1, 2, 3, 4 in building 8/9 at LD8. This has resulted in a loss of power for multiple customers and IBX Engineers are working to resolve the issue.
The next update will be provided in approximately 30 mins.
[Update 9:50]
We have seen that some connections are returning and we expect this to continue – The fire alarm has now been re-instated which allows the Data Centre Staff to work round the issue with the UPS at the site. We are still seeing some links down but at this point in time this is not customer affecting
We will update further as we have it
We will be upgrading some of our core routers that terminate broadband sessions later this evening – each will be down for less than 30 seconds for a firmware upgrade. These will be staggered over the evening. Most affected customers will reconnect within seconds automatically. if you have any issues please reboot your router and if that fails for any reason, please call into or email support in the normal way.
This only affects ADSL, FTTC & FTTP lines.
15:42
20/12/18 15:00 UPDATE
The new UPS system has now been installed and commissioned. We now have a full N+1 resilient UPS system in our Avro Court Data Centre. Power is no longer at risk.
19/12/18 10.32am UPDATE
The USP manufacturer has been on-site yesterday and today. After discussions with them, we have decided to take this opportunity to upgrade the current UPS to a later model, giving us increased cover and reliability. This is due to arrive on-site at some point today. Installing, commissioning and testing will however likely roll into tomorrow.
The power supply to the data centre continues to be at risk until this replacement work is completed.
18/12/18 12:10pm UPDATE:
The engineer has sourced the necessary spares and is again en-route to site. We expect that this work will take the rest of the afternoon with the necessary tests being run and, assuming all goes to plan, the UPS will be brought back on-line during the early evening.
The power supply will be at risk until this is completed.
UPDATE:
The UPS is still being worked on. Whilst we expect this to be completed today, power supply to the data centre should still be considered at risk until this is complete.
We had a planned service today of our UPS, by specialist engineers. This has uncovered a problem that has caused our main UPS to not come online fully. Currently services in the Data Centre are working fine but the power should be considered at risk until the UPS is fully restored to service.
The Specialist engineers are currently arranging the parts required and once here they will be swapped out. We will update this post as we have more details
As part of our contract with our specialist contractors the UPS in Avro court is having a regular service on 18/08/18.
This involves checking the UPS is functioning correctly, batter health etc
This should not cause any interruption to service but as work is being undertaken here there is always a very small risk.
Any work that is deemed to have a potential of interruption will take place after 5pm. Our own engineers will be on site throughout to monitor the work
We will update this as the work completes