Virgin Leased Line instability

We are seeing a small number of Virgin Media leased lines have intermittent drops – Virgin Media have located a network event. Their latest update is:

Our Network Teams have carried out investigations and they suspect the issue is due to a fibre
disturbance between Peterborough and Leicester causing the intermittent network connectivity issues.

Next steps consist of intrusive power level testing and an emergency change will be scheduled for the works activities.

Services have remained stable since 13:21 however health checks are in place to monitor services.

We’ll provide further updates as soon as we get feedback.

We are also seeing services stable but will continue to update as we have them

At Risk Planned Work – UPS Maintenance Huntingdon 5/7/2024

During the day of 5th July we will have specialist engineers working on our UPS system at our Huntingdon Data Centre.

This should not impact services at all as we have a N+1 redundant system. However there is always a slight risk and we will have slightly lower redundancy during the work

We will update this as the work progresses and completes

At Risk Planned Work – Generator Maintenance Huntingdon 22/7/2024

During the day of 22nd July we will have specialist engineers working on our Generator system at our Huntingdon Data Centre.

This should not impact services at all as as even in the case of a power failure we have a UPS system. However there is always a slight risk and we will have slightly lower redundancy during the work

We will update this as the work progresses and completes

OUTAGE – Telehouse North [resolved]

We are aware of an outage affecting our core switch in Telehouse North. This initially was a wider issue however we have now isolated this issue to services directly connected in Telehouse North.

We have an engineer en route (eta should be less than 30 minutes) and we will then have a clearer picture of the cause – and hope that service will be restored shortly thereafter

We will update this as soon as we have it

[resolved]
This issue was fully by 17:35 – One of our switches in Telehouse North had crashed – the route cause will be investigated and worked on if needed. We apologise for the issue seen here

Virgin Media Bearer Down 20/5/2024

We have seen one of our Virgin Media bearers drop this morning which has affected a number of our leased lines. This has been reported to Virgin who are actively investigating this.

We are sorry for any issues seen and will update this case within 1 hour

Router Reboot / Upgrades 27 and 28th April 2024 [completed]

We have seen one of our core IBR routers crash this afternoon which affected reachability for a subset of customers

Service is now back however we would like to upgrade and reboot the affected devices later this evening to bring them to the latest code and reduce the risk of any further issues.

There are currently four routers left from the upgrade process mentioned in our previous post – we will aim to complete the upgrade and reboot them later this evening and if necessary tomorrow evening – though we hope to complete the work in one evening.

This should have minimal impact to customers as traffic should just route via other core routers but it is obviously ‘at risk’

[update 28/04 9AM]
All work on this completed overnight and we saw very minimal if any impact to customers. All core routers are upgraded and working as expected. If you see any issues related to this please do contact support via the normal routes

 

Planned Work – Router Upgrade Project [completed]

We are planning to start a rolling project of upgrading our core routers over the next few weeks – this will involve the LNS routers (those that Broadband customers connect to) and our Internet facing border routers.

In the case of the LNS / Broadband routers this will lead to a brief drop and re-connect of your service when the router is rebooted after the update.

In the case of the core routers this should not be visible as in most cases traffic will simply re-route round them and we will remove traffic from the router we are working on before upgrading it

We will update this posy shortly with a schedule and update on the project

[completed]

All upgrades have been completed for this work.

 

 

OUTAGE: Some Virgin Media Based Leased Lines 20/3/2024 Update 2

We are seeing one of our Virgin Media NNIs down currently affecting services that are provided on the affected NNI

This is being raised with Virgin Media currently and we will update this post as we have mode details.

NOTE this only affects leased lines on the one affected link – other services are working.

[Update 13:10]

We have the following update from Virgin Media

We’re aware of an issue impacting your servicesNetwork Teams have identified a number of 10gb High Capacity Services have lost connectivity.An Engineer had already been requested and arrived onsite in Northampton at 12:13 and investigations are underway. All respective support teams have been engaged to progress investigations and create a fix plan.

We will update as we know more

[Update 14:00]

We have seen the affected NNI return to service just before 2pm. We do not have an update from Virgin to confirm that this is fully resolved – but for now we see services  back to normal. We do treak them as ‘at risk’ until the issue is closed down.

Planned Work – At Risk – Huntingdon and Manchester 19/3/2024

We have been advised by Virgin Media they will be performing work on a backup link between our Huntingdon and Manchester Data Centre.

The work is scheduled for the early hours of the 19th March.

We do not anticipate any downtime from this but the network does have reduced resilience while this work is being undertake,

 

Planned Word – Router Upgrade / Reboot – 17/1/24

Due to a warning on one of our Broadband Routers in Telehouse West we may been to upgrade this and reboot it later this evening.

If your session is affected your session is affected you should fine you re-connect within a few minutes

If for any reason the service continues to be down – please then reboot your router

We will update this after the work completes – Sorry for any issues