At Risk – Slough Data Centre [updated]

One of the two links to our Slough Data Centre is currently down – it dropped overnight.  The Data Centre has dual resiliant links so at this point service is not affected – however services at this Data Centre should be considered at risk.

The team have reported this issue to the fibre provider and we will update as we hear back

We will update this by 10am at the latest

[Update 8:10am]
Our fibre provider have acknowledged our case and have dispatched engineers to one of the Data Centres – current ETA for them to arrive is just after 10am.

[update 14:00]
Our Fibre Provider has completed OTDR testing on the fibre span and can see no fault on the fibre span – but have stated that the Data Centre Cross connect may be at fault. As such a case has been raised to have this checked. We will update on this as soon as we have it

Loss of Resilience – Manchester and Huntingdon

We are seeing one of the core links down between Manchester and Huntingdon currently. This seems to be due to a known fault on the Virgin Media Network.

Virgin are working to resolve this issue. At this point this has no affect on our services – however there is a loss of resilience to these two sites currently as a result

We will update as we hear more

Virgin Leased Line instability [resolved]

We are seeing a small number of Virgin Media leased lines have intermittent drops – Virgin Media have located a network event. Their latest update is:

Our Network Teams have carried out investigations and they suspect the issue is due to a fibre
disturbance between Peterborough and Leicester causing the intermittent network connectivity issues.

Next steps consist of intrusive power level testing and an emergency change will be scheduled for the works activities.

Services have remained stable since 13:21 however health checks are in place to monitor services.

We’ll provide further updates as soon as we get feedback.

We are also seeing services stable but will continue to update as we have them

At Risk – Fibre Down – 10/5/23 [resolved]

We have had an alert of one of our fibre links between Cloud House and Telehouse is currently down, At this point it is not affecting the merula Core netwoek but does mean that we have a loss of resiliance between sites.

This is service impacting to customers taking a wave service between these two sites

The issue has been raised to the fibre provider who are actively working on the issue

We will update as we know more here

[update 11/5/23 8:00]

The fibre provider found the cause of the break and restores service on this link approx 1:45am. We have been monitoring this link since and all appears stable.

 

At Risk Huntingdon Data Centre – 23/9/22 [completed]

As we have previously announced due to the closure of Equinix Reynolds House in Manchester we have moved our Manchester PoP to a new location

This Friday our carrier is moving our 10G Link from Huntingdon to Manchester to the new PoP in Manchester

Through this we will be using our primary link to London so we do not expect any issues – However for a short period we will be without redundancy / backup to Huntingdon

Note in this case the work is through the daytime – as his means additional engineering resources are available to ensure the change is smooth and without issues

We will update this post as the work progresses and completes

[Completed]
Virgin Media completed the circuit move Friday evening – we now have moved all links back to our net resilient network core.

At Risk – Network Changes 6/8/22 – 7/8/22 [update 1]

As part of our plan to enhance our network capacity and resilience over the next few days we will be migrating a number of core links to a new model of EPVN. The new config will be in place before any old links are removed. However there is a risk through this for brief drops and routing changes being visible.

This has already been tested in our lab and the configuraton is in place the final changes are just to move from the old configuration to the new setup

We apologise for the short notice for this work

We will update as the work progresses

[Update 1]
The work on the IBR core routers completed yesterday bringing our new Manchester core routers all live. There is some work this evening on our Juniper Core routers – again we don’t expect any substantal issues but you may notice some routing changes and bried periods of packet loss as the new routing converges