At Risk Network Change – 25/09/2022

Through a customer report we have become aware of an issue with a MPLS link across our network. To resolve this we need to change the MTU across some of our core links.

We need to make this change this evening to fully restore service on the link. We will make the change link at a time – Due to the resilience built into our network we don’t anticipate any issue on our network – However with any config change there is a risk of a small number of very brief dropouts or you may see slightly different routing between sites.

We will update this as the work completes

 

 

At Risk Huntingdon Data Centre – 23/9/22 [completed]

As we have previously announced due to the closure of Equinix Reynolds House in Manchester we have moved our Manchester PoP to a new location

This Friday our carrier is moving our 10G Link from Huntingdon to Manchester to the new PoP in Manchester

Through this we will be using our primary link to London so we do not expect any issues – However for a short period we will be without redundancy / backup to Huntingdon

Note in this case the work is through the daytime – as his means additional engineering resources are available to ensure the change is smooth and without issues

We will update this post as the work progresses and completes

[Completed]
Virgin Media completed the circuit move Friday evening – we now have moved all links back to our net resilient network core.

Switch Replacement – 10/09/2022 22:00 – 11/09/2022 03:00 [completed]

Following on from the switch replacement in London Harbour Exchange square recently the vendor had advised us of a potential fan issue on the switch supplier and have supplied is with a second replacement switch,

We will swap this switch (again) on Saturday evening to ensure that all fans etc are fully working

While we have allowed a four how window here we expect the outage to be no longer than 10-20 minutes all being well.The new switch is all pre-configured so this is a simple chassis swap

We apologise for the extra work here but we would prefer to ensure there’s no potential unplanned issue here

This work was completed at 22:16 on Saturday with an outage of less than 10 minutes

At Risk – Huntingdon Data Centre 1/9/2022

We have been infirmed that Virgin media are working on one of the 10G Links to our Huntingdon Data Centre

Planned Work Start (Local UK): 01/09/2022 00:01
Planned Work End (Local UK): 01/09/2022 06:01

Impact: There will be a loss of service at some point during the timeframe stated above whilst we complete the maintenance work.

The service outage should only last for up to 4 hours; however a 6 hour timeframe has been given in the event of any unforeseen circumstances.

This Emergency planned work is required to replace a hard spliced fibre patch at our site in Northampton that is causing degraded services
Corresponding Fault ticket reference = F010066288.

We do have resilient routes to the data centre however the data centre should be deemed to be at risk during these works

At Risk – Huntingdon Data Centre 29/08/2022

Virgin Media have some planned work over the next few days which may affect one of the two core links to our Huntingdon Data Centre. We have spent the last few weeks improving and strengthening our network with a new EVPN based core.

We had planned a full failover test to be announced for late next week – however with this planned work – we have decided to run at least one part of the test later today.

In theory this will only leave a drop of a few seconds as traffic will re-route – However there’s a risk if there’s any issues some connectivity to Huntingdon MAY be lost while we troubleshoot any issues. We will have full control of this and can easily restore service if we need more time to troubleshoot

We are sorry for the short notice here but we’d rather be prepared and happy that failover works exactly as it is designed – as opposed to finding this out in a real issue on a link.

We will update as this work progresses / completes

 

Switch Replacement – 26/08/2022 22:00 – 27/08/2022 03:00

Further to the work overnight the issue on our Core Harbour Exchange Switch is still not allowing config changes to be committed. At this point this is not causing any network issues – However this is obviously causing a risk and it means that we cannot provision new services on it, The errors mean there is a risk of an unexpected failure.

We have a replacement switch from the vendor which is  being pre-configured during the day today

Later this evening we will swap the switch chassis from the old to the new one, This will cause some network disruption as traffic re-converges. While the switch is down services terminating in Harbour Exchange will  be down and this will also affect one route into our Huntingdon Data Centre

We anticipate the outage to be approx 30-60 minutes

We are sorry for this short notice – however we wanted to ensure the issue was resolved before any customer facing issues occurred

Virgin Media Leased Lines :: Planned works 26/8/2022

We have been advised of Emergency planned works affecting a number of lines this evening:

Planned Work Start (Local UK): 26/08/2022 00:01
Planned Work End (Local UK): 26/08/2022 06:00

Impact: There will be a loss of service at some point during the timeframe stated above whilst we complete the maintenance work.

The service outage should only last for up to 1 Hour, however a 6 hour time frame has been given in the event of any unforeseen circumstances.

This is for emergency planned work to repair a fault on the network in the Papworth area. Resolution for fault reference: F010061503

Customers on  the lines Virgin have listed have been notified.

At Risk :: Switch Maintenance – 26/08/2022 0:01 – 02:00

Our Juniper Switch in London Harbour Exchange Square is showing an error when any new config is committed to it.

Juniper have provided their suggested work-around/fix for this and we will perform this just after midnight tonight.

There should be no impact from this work at all – However we would rather announce this as at risk in case of any unexpected impact

We will update this once the work completes

[update 26/08/22 8:30]
The work last night while not causing any issue did not resolve the issue, The vendor has supplied a replacement Core Juniper switch and this will be swapped out later tonight

Planned Work – Virgin Media [05/09/2022]

Virgin Media are performing a network upgrade on their code network between  midnight and 6am. There’s an expected downtime of 5 minutes. This has been reported to affect a small number of our lines on both our London bearer or terminated in London.

From their announcement


There will be a loss of service at some point during the timeframe stated above whilst we complete the maintenance work.

The planned outage is expected to occur between 00:01am and 02:00am and should only last for up to 30 minutes or if notified for 5 minutes where this is for Network Re-convergence only. However, should we incur any card failure and a replacement is required then as a worse case we will need to use the 6 hour change window.

This work is to upgrade software on equipment on our Network at multiple sites.


 

We will monitor these lines and ensure that they restore during the announced window

We apologise for any issues from this work

Planned Work – Virgin Media [09/09/2022]

Virgin Media are performing a network upgrade on their code network between  midnight and 6am. There’s an expected downtime of 30 minutes. This has been reported to affect a small number of our lines on both our London bearer or terminated in London.

From their announcement


There will be a loss of service at some point during the timeframe stated above whilst we complete the maintenance work.

The planned outage is expected to occur between 00:01am and 02:00am and should only last for up to 30 minutes or if notified for 5 minutes where this is for Network Re-convergence only. However, should we incur any card failure and a replacement is required then as a worse case we will need to use the 6 hour change window.

This work is to upgrade software on equipment on our Network at multiple sites.


 

We will monitor these lines and ensure that they restore during the announced window

We apologise for any issues from this work