Virgin Leased Line issues 18/12/2024 [resolved]

Dec 18, 2024 | Outages

We have become aware during the day of an increasing number of drop outs on one of our Virgin Media Bearers which provides service for approx 30 leased lines. Virgin are aware of this and are actively working to identify and resolve the fault. Note this does not affect all lines  just those terminating on the affected handover in Huntingdon. Note lines are not hard down but we are seeing multiple drops for 30-90 seconds.

We will update this as we have a further update

[Update 22/12/2024]
Virgin Media have been working to resolve this issue – however we are still seeing ongoing drops on this line. There was a longer outage earlier today to swap out a number of cross connects and cards – however we are still seeing very brief drops on this link. Virgin Media are performing more work this evening to investigate the cause here and we will update more as soon as we know more.

[Update 23/12/24]
Virgin replaced some kit in one of their POPs last night however the issue is still ongoing with 5-10 second drops on links on one of our backhaul links (note this only affects a sub-set of our circuits). Virgin have asked for an extended downtime to perform a full end to end test. Given this issue only occurs every hour or so the downtime is likely to be 2-3 hours and will take place from 10pm this evening. We have also asked Virgin Media if we can move some of the single homed lines to an alternative handover while this is investigated. We will update further as we have it

[update 24/12/24 – 9:30]
Virgin completed their work overnight – however the issue still occured after this – this has been escalated back to Virgin Media who are currently working on the next steps to restore stability. We will have a further update by 11:30am

[update 24/12/24 11:00]
We are aware these lines have just dropped and are awaiting an update from Virgin. We are aware they are looking to revert last nights change given that this has not resolved the issues. We have pushed for an update and will update this as soon as we have it. As I type this I can see the service has just come back – we will update this further as we have it. Expect the next update by 13:00 at the latest

[update 24/12/24 13:00]
Virgin Media have moved this circuit to a new card / hardware just before 11am and then reverted last nights change at 11:50 which caused the longer drops this morning, We have not seen any further drops seen but both Virgin and ourselves are keeping a close eye on this. We will update again by 16:00 if there is no news or developments before then

[update 24/12/24 13:20]
Virgin have now confirmed they have completed their changes on this line for now and they and we are monitoring this for stability. We will update at 18:00 if there are no changes or drops. Virgin are keeping th fault open for a number of days to ensure this stays stable

[update 24/12/24 16:20]
The link has remained stable since just before midday. Both Virgin and ourselves continue to monitor this closely for any further drops. We will update this by 20:00 unless there are any significant changes

[update 24/12/24 19:00]
The link has remained stable since just before midday. Both Virgin and ourselves continue to monitor this closely for any further drops. We will not provide any further updates until 27/12 unless there is any significant change however we will be monitoring the link closely

[update 27/12/24 07:00]
The link has remained stable since the change by Virgin Media just before midday on the 24th. We are waiting for the RFO from Virgin but believe this issue is now resolved. Sorry for any issues this may have caused

OUTAGE – Telehouse North [resolved]

May 30, 2024 | Outages

We are aware of an outage affecting our core switch in Telehouse North. This initially was a wider issue however we have now isolated this issue to services directly connected in Telehouse North.

We have an engineer en route (eta should be less than 30 minutes) and we will then have a clearer picture of the cause – and hope that service will be restored shortly thereafter

We will update this as soon as we have it

[resolved]
This issue was fully by 17:35 – One of our switches in Telehouse North had crashed – the route cause will be investigated and worked on if needed. We apologise for the issue seen here

OUTAGE: Some Virgin Media Based Leased Lines 20/3/2024 Update 2

Mar 20, 2024 | Outages

We are seeing one of our Virgin Media NNIs down currently affecting services that are provided on the affected NNI

This is being raised with Virgin Media currently and we will update this post as we have mode details.

NOTE this only affects leased lines on the one affected link – other services are working.

[Update 13:10]

We have the following update from Virgin Media

We’re aware of an issue impacting your services
Network Teams have identified a number of 10gb High Capacity Services have lost connectivity.
An Engineer had already been requested and arrived onsite in Northampton at 12:13 and investigations are underway.
All respective support teams have been engaged to progress investigations and create a fix plan.

We will update as we know more

[Update 14:00]

We have seen the affected NNI return to service just before 2pm. We do not have an update from Virgin to confirm that this is fully resolved – but for now we see services  back to normal. We do treak them as ‘at risk’ until the issue is closed down.

Brief issue – Huntingdon 4/6/2023 – RESOLVED

Jun 4, 2023 | Outages

We saw loss of connectivity to some of the servers in Huntingdon between 3pm and 4pm this afternoon. This was due to an issue with one power feed to a core rack.

Most of our new network is now dual fed- however some of the legacy network on the older Cisco switches is single fed. We saw one switch down during the issue.

Our engineers attended site and resolved the issue – we are performing further checks and will continue the replace the older switches as quickly as we can.

We apologise for any issues