Switch Reboot Telehouse North and East [10/10/2022]

Following the advice of Juniper we are deploying an upgrade to the Juniper Software to our core switch in telehouse North and East and rebooting to clear an at risk issue on the switch.

Expected downtime per site is approx 10 minutes – we will update the sites one at a time.

While the switches are working find now we the upgrade clears a known issue that can cause instability which we have seen in the Lab.

 

At Risk Network Change – 25/09/2022

Through a customer report we have become aware of an issue with a MPLS link across our network. To resolve this we need to change the MTU across some of our core links.

We need to make this change this evening to fully restore service on the link. We will make the change link at a time – Due to the resilience built into our network we don’t anticipate any issue on our network – However with any config change there is a risk of a small number of very brief dropouts or you may see slightly different routing between sites.

We will update this as the work completes

 

 

At Risk Huntingdon Data Centre – 23/9/22 [completed]

As we have previously announced due to the closure of Equinix Reynolds House in Manchester we have moved our Manchester PoP to a new location

This Friday our carrier is moving our 10G Link from Huntingdon to Manchester to the new PoP in Manchester

Through this we will be using our primary link to London so we do not expect any issues – However for a short period we will be without redundancy / backup to Huntingdon

Note in this case the work is through the daytime – as his means additional engineering resources are available to ensure the change is smooth and without issues

We will update this post as the work progresses and completes

[Completed]
Virgin Media completed the circuit move Friday evening – we now have moved all links back to our net resilient network core.

Switch Replacement – 10/09/2022 22:00 – 11/09/2022 03:00 [completed]

Following on from the switch replacement in London Harbour Exchange square recently the vendor had advised us of a potential fan issue on the switch supplier and have supplied is with a second replacement switch,

We will swap this switch (again) on Saturday evening to ensure that all fans etc are fully working

While we have allowed a four how window here we expect the outage to be no longer than 10-20 minutes all being well.The new switch is all pre-configured so this is a simple chassis swap

We apologise for the extra work here but we would prefer to ensure there’s no potential unplanned issue here

This work was completed at 22:16 on Saturday with an outage of less than 10 minutes

At Risk – Huntingdon Data Centre 1/9/2022

We have been infirmed that Virgin media are working on one of the 10G Links to our Huntingdon Data Centre

Planned Work Start (Local UK): 01/09/2022 00:01
Planned Work End (Local UK): 01/09/2022 06:01

Impact: There will be a loss of service at some point during the timeframe stated above whilst we complete the maintenance work.

The service outage should only last for up to 4 hours; however a 6 hour timeframe has been given in the event of any unforeseen circumstances.

This Emergency planned work is required to replace a hard spliced fibre patch at our site in Northampton that is causing degraded services
Corresponding Fault ticket reference = F010066288.

We do have resilient routes to the data centre however the data centre should be deemed to be at risk during these works

At Risk – Huntingdon Data Centre 29/08/2022

Virgin Media have some planned work over the next few days which may affect one of the two core links to our Huntingdon Data Centre. We have spent the last few weeks improving and strengthening our network with a new EVPN based core.

We had planned a full failover test to be announced for late next week – however with this planned work – we have decided to run at least one part of the test later today.

In theory this will only leave a drop of a few seconds as traffic will re-route – However there’s a risk if there’s any issues some connectivity to Huntingdon MAY be lost while we troubleshoot any issues. We will have full control of this and can easily restore service if we need more time to troubleshoot

We are sorry for the short notice here but we’d rather be prepared and happy that failover works exactly as it is designed – as opposed to finding this out in a real issue on a link.

We will update as this work progresses / completes