Mar 13, 2022 | Planned Work
We have been informed that Virgin Media will be performing a re-route on one of the backhaul links between London and our Huntingdon Data Centre between the 24/3/2022 and 25/3/2022
We have resilient links to provide service here however this will reduce the resilience for services in Huntingdon and also reduces resilience to Manchester
The planned work has a 10 hour window however we don’t anticipate the link being down that long.
We will update this post if we see any impact and once the line is restored.
Mar 13, 2022 | Planned Work
During Monday 14th March our specialist contractors will be replacing the batteries on our UPS in our Huntingdon Data Centre.
While this should have no impact on services – it does pose a slight risk and reduces resilience to services in our Huntingdon Data Centre
We will update as this work progresses
[completed 14:00]
The battery replacement is completed and the UPS and resilience are now back to normal in our Huntingdon Data Centre
Mar 10, 2022 | Unplanned downtime
Unrelated to the issue yesterday – one of our Juniper switches in Telehouse North is reporting errors and is causing routing to toggle between two routes which is causing some latency issues etc.
Under recommendation from our vendor we will reboot the routing engine between 10pm and midnight this evening.
This will affect customers directly connected and will mean that some Broadband customers who terminate in Telehouse may see their connection drop and re-connect to an alternative LNS
We will update this post once the work completes. We are sorry for the short notice but would rather resolve this now before there is any risk of an unplanned outage as a result
Mar 9, 2022 | Information
We are currently seeing an issue with some traffic routing via Telehouse West – we currently believe one of our core switches has a forwarding issue,
Our NOC team are currently working to resolve this which MAY invoice a brief reboot of the affected switch – causing a short downtime for services directly connected to that switch
We apologise for the brief and short notice reboot but want to resolve the issue as quickly as we can
[Update]
The switch is now back from the reboot and we are seeing ttraffic and routing back as it should be – we will investigate further the cause here as a background task to ensure it does not re-occur
Feb 14, 2022 | Planned Work
During the 15th February 2022 engineers will be replacing the fuel on our generator at our Huntingdon Data Centre.
This will not cause any direct issues and we still have our UPS system to maintain power to the Data Centre. It does however mean that in the event of a power failure on the Utility supply there is a risk as for a short period the Generator will not be available to supply load.
Our suppliers will be working hard to ensue that the at risk period is kept to an absolute minimum
NOTE the new fuel will be the more environmentally friendly HVO fuel as part of our commitment to reduce the environmental impact of our company
We will update this post as the work progresses
[Update]
This work completed this morning and the generator is back available as needed
Feb 11, 2022 | Unplanned downtime
We are aware of a Single VM Host currently down which is being worked on now. Our Engineers are migrating the Virtual Machines to an alternative host. We will update this post within 30 minutes with an update
Just an update – this should not affect most users – unless we host a site for you on this particular server. We are currently migrating the data and drives to new hardware and expect to have this back up and running very shortly
The Affected host is now back up and running – the Virtual Machines have been moved to alternative hardware and we see all servers now running as they should. We are sorry for any issues this may have caused.
Feb 1, 2022 | Planned Work
Following on from the fibre break on our Huntingdon to Manchester link yesterday our supplier is carrying out further work overnight to replace approx1km of fibre following the damage they found. This work is currently scheduled to run between 11pm this evening until approx 7am on Wednesday.
We are told the link should not be down for the entire duration but will be down for some point of this.
As previously this will not cause a outage on our network but it does reduce the resilience – and therefore create risk for connectivity at our Huntingdon Data Centre and Manchester.
We will update this post as the work starts and as it progresses / completes
Jan 31, 2022 | Information
We are aware of one of the two 10G Link lines to Huntingdon down (this is not the link worked on over the weekend)
We have a case raised to our supplier and are waiting for updates. This is not service affecting but removes resilience for our Huntingdon Data Centre and our Manchester PoP.
We will update this as we know more
[Update 15:30]
Virgin Media have confirmed this is the result of a fibre break in Leicester. This is being worked on and we will update you as we hear. As before this is not currently affecting service but does leave our service at risk.
[Update 19:30[
The link came back approx 17:15 and Virgin have now confirmed they located damaged fibre which has now been re-spliced. Virgin have now closed the fault. We will continue to monitor however we do not anticipate any further impact
Jan 26, 2022 | Uncategorized
We have been informed that Virgin Media will be performing a re-route on one of the backhaul links between London and our Huntingdon Data Centre between the 28/1/2022 and 29/1/2022
We have resilient links to provide service here however this will reduce the resilience for services in Huntingdon and also reduces resilience to Manchester
The planned work has a 10 hour window however we don’t anticipate the link being down that long.
We will update this post if we see any impact and once the line is restored.
Note this is a re-schedule of the earlier announced work
[Update 29/1/2022 9:30]
We saw the link drop approx 12:30am and returned approx 5:30. Failover went as planned and there was no noticeable affect on traffic
Jan 11, 2022 | Uncategorized
We saw a drop in most traffic to transit providers between 19:15 and 19:40 this evening.
This showed as a drop in traffic over that period for any traffic via Transit (connectivity reached over our own peering was fine.The effect would have been the inability to reach certain sites dueing this time.
The issue resolved itself while we started investigating. We believe this issue was caused by an upstream network as opposed to something inside our own network
We are closely monitoring and investigating this issue and working with our upstream providers. As and when we hear back we will update this post
