OUTAGE: Some Virgin Media Based Leased Lines 20/3/2024 Update 2

We are seeing one of our Virgin Media NNIs down currently affecting services that are provided on the affected NNI

This is being raised with Virgin Media currently and we will update this post as we have mode details.

NOTE this only affects leased lines on the one affected link – other services are working.

[Update 13:10]

We have the following update from Virgin Media

We’re aware of an issue impacting your servicesNetwork Teams have identified a number of 10gb High Capacity Services have lost connectivity.An Engineer had already been requested and arrived onsite in Northampton at 12:13 and investigations are underway. All respective support teams have been engaged to progress investigations and create a fix plan.

We will update as we know more

[Update 14:00]

We have seen the affected NNI return to service just before 2pm. We do not have an update from Virgin to confirm that this is fully resolved – but for now we see services  back to normal. We do treak them as ‘at risk’ until the issue is closed down.

Planned Work – At Risk – Huntingdon and Manchester 19/3/2024

We have been advised by Virgin Media they will be performing work on a backup link between our Huntingdon and Manchester Data Centre.

The work is scheduled for the early hours of the 19th March.

We do not anticipate any downtime from this but the network does have reduced resilience while this work is being undertake,

 

Planned Word – Router Upgrade / Reboot – 17/1/24

Due to a warning on one of our Broadband Routers in Telehouse West we may been to upgrade this and reboot it later this evening.

If your session is affected your session is affected you should fine you re-connect within a few minutes

If for any reason the service continues to be down – please then reboot your router

We will update this after the work completes – Sorry for any issues

 

At Risk – Switch Maintenance HEX 7/1/24

We are seeing a minor error on one of our switches in HEX for the past few weeks. We have tried a number of non service affecting fixes – which have unfortunately not cleared the issue.

The warnings at this point are not service affecting – however we want to clear this before it becomes a issue.

We are in the data centre for some other work tomorrow – and if time allows we will run some checks while we have an engineer there.

There is a risk we may need to power cycle the switch to try and clear the issue. Depending on what we find we may perform this tomorrow or we may schedule this another time.

There is no core routing via this switch however there are links to some leased line and broadband connections on the switch – IF the switch is rebooted most of these will see a brief drop while the sessions move to alternative links

We will update this as we know more

[update]

We performed some initial work here but will re-plan the re-boot / replacement for another date.

At Risk – Some Broadband Sessions – 21/11/2023

One of our wholesale partners are upgrading the switch that one of our feeds is connected to over night on the 21st November

We will gracefully move sessions to the second NNI before the work starts and re-enable the affected link after their work.

Customers with sessions on the affected link may see their connection drop briefly when this work takes place

At Risk – Core Link down – Loss of Resilience 15/11/2023 [resolved]

Our provider for one of our core links between London and Manchester has a fibre break meaning that our core link between London and Manchester is currently down.

This is not currently causing any issue but it does mean that we have a lack of resilience to both the Manchester and Huntingdon sites.

Our supplier have identified the fibre break location caused by a digger and are working to repair the damage currently.

We will update this as we know more

[resolved[

The fibre damage was resolved yesterday and the link has remained stable since

 

At Risk Planned Work – Generator Maintanance Huntingdon 21/7/2023

During the day of 20th July we will have specialist engineers working on our Generator system at our Huntingdon Data Centre.

This should not impact services at all as as even in the case of a power failure we have a UPS system. However there is always a slight risk and we will have slightly lower redundancy during the work

We will update this as the work progresses and completes

At Risk Planned Work – UPS Maintanance Huntingdon 20/7/2023

During the day of 20th July we will have specialist engineers working on our UPS system at our Huntingdon Data Centre.

This should not impact services at all as we have a N+1 redundant system. However there is always a slight risk and we will have slightly lower redundancy during the work

We will update this as the work progresses and completes

Customer Leased Line Router Upgrades – 19/7/23 – 21/7/23

Our router vendor used for customer leased line routers has released a number of firmware updates. Over these three evenings we will upgrade the firmare on most customer routers. Each router will see aprrox 5 minutes downtime while the router reboots after the upgrade.

We will confirm as the work progresses and completes

At Risk – DWDM Update – 17/7/23 [completed]

We are planning a small upgrade to our links between Telehouse North and Telehouse West and Telehouse Noth and harbour Exchange quare on the evening of Monday 17th July

This will leave the sites at risk and some potential very brief blips in connectivity as the fibres on these buildings are moved over to our new DWDM platform.

We will update this post once the work completes

[completed]
This work completed as planned with minimal impact – the DWDM fibre links are now working as expeted