Emergency work 15/02/2018
Our supplier has identified a potential issue with a fibre bearer which holds some backup links to our Huntingdon Data Centre. They are making a config change to correct this overnight.
As these are backup links it should not have a significant impact but there is some chance of packet loss as the link drops / returns as the network re-converges.
We will update this post as the work progresses
Planned Work – 9/2/2018 10PM
We have noticed a small issue on a switch port connected to some leased lines. The issue is not currently service affecting – and needs a simple one line config change to resolve.
As there is a very small risk to a few seconds of packet loss during the change this will be performed tonight
Only leased lines connected to a single bearer in Telehouse North will be affected and we don’t expect more than a few seconds of packet loss during the change
Any questions or issues please contact support via the normal means
PLANNED WORK: Core switch upgrades.
PLANNED WORK: 1/2/18 UPDATE
Further to the outage yesterday and the corrective work by the carrier we will attempt to bring the link that failed back into service this evening. If all works to plan there may be a brief network disruption while the network reconverges. If the link fails again we will remove it from the network and continue work with our supplier.
The link was brought on line approx 12:30am this morning – However the previous issue re-occurred. This has been raised to the supplier who believe this is resolved this correctly now, We will plan a further maintenance shortly to bring the link back into service. In the mean time there is no reduction to service – but there is a slight reduction in resilience to our Manchester PoP
OUTAGE: Network drop 31/1/18
UPS & Generator Maintenance – Avro Court
Our specialist contractors will be working on the Generator and UPS at Avro court late this afternoon for the regular maintenance. This is part of our plan to ensure that the best service is available to customers.
While this work should not have any affect on our services there is a reduction of resilience during the work and there are engineers working on them so there is a very low risk of this affecting our services.
We will update this post once the work completes
COMPLETED: possible ADSL/FTTC outages: THE maintenance 25/11/17
UPDATE: This work is completed
UPDATE: this should have read THE, apologies.
We are planning a UPS replacement in our data centre in THE following recent problems caused by it, as the rack appears to be at risk of a power loss whenever we work near it. We have a replacement and we plan to swap to that and then investigate at a later date the cause of the issue on the current UPS.
This will mean a brief downtime of some hardware in the rack. Currently only ADSL/FTTC lines will be customer affecting there, which should just drop and then auto connect at another of our POPS. This will happen during the course of Saturday evening.
If after this work is completed and a router reboot, you’re still unable to connect, please contact support in the usual way.
COMPLETED: last Merula mail server upgrades
The email servers are being upgraded over the weekend starting late on Friday evening to a pair of faster mirrored servers. This won’t affect everyone but during the course of this migration email will appear to have disappeared from in-boxes but this isn’t any cause for concern and is merely a side-effect of the migration process.
No email will be lost; it’s just the final synching process needed to improve the mail-servers for everyone. During the course of the weekend the email should fully re-appear.
New email will be continue to be delivered straight away – older emails will be restored as the sync process progresses. Any security certificate error messages can be safely ignored for the duration of this migration. These will disappear after both servers are finally fully synced
thos is work is completed if there are any issues with your mail please contact support via the normal routes ASAP.
Merula mail server upgrades
The email servers are being upgraded over the weekend to a pair of faster mirrored servers. During the course of this migration email will appear to have disappeared from in-boxes but this isn’t any cause for concern and is merely a side-effect of the migration process
No email has been lost – it is in the process of being synced. During the course of the weekend the email should fully re-appear.
New email will be delivered instantly – older emails will be restored as the sync process progresses. Any security certificate error messages can be safely ignored for the duration of this migration. These will disappear after both servers are fully synced and updated.
We will confirm when this process has completed.
UPDATE 2 – HEX UPS Maintenance [23/09/2017]
Our in rack UPS in our Harbour Exchange PoP is showing a possible battery failure. We have a replacement batteries and will be replacing these tomorrow (23/09). These can be swapped without removing power. However with any activity there is a small risk of disruption to services connected to this rack
UPDATE: It appears there may be a more serious issue with the UPS than failed batteries. A short time ago we lost all Power in our HEX PoP which caused a period of Network instability
We believe that all Network access with the exception of services directly connected to HEX is now up. We are working on HEX to bring service back asap
UPDATE 13:50 The Data centre have located a trip has failed which has lead to no Power being available to our rack. The Data Centre technicians are working to restore power to us as a priority. We expect a further update very shortly