At Risk – Network Changes 6/8/22 – 7/8/22 [update 1]

As part of our plan to enhance our network capacity and resilience over the next few days we will be migrating a number of core links to a new model of EPVN. The new config will be in place before any old links are removed. However there is a risk through this for brief drops and routing changes being visible.

This has already been tested in our lab and the configuraton is in place the final changes are just to move from the old configuration to the new setup

We apologise for the short notice for this work

We will update as the work progresses

[Update 1]
The work on the IBR core routers completed yesterday bringing our new Manchester core routers all live. There is some work this evening on our Juniper Core routers – again we don’t expect any substantal issues but you may notice some routing changes and bried periods of packet loss as the new routing converges

At Risk – Generator Service – Huntingdon Data Centre – 15/07/2022

Our Supplier Engineers will be performing their regular service on our Generator system at our Huntingdon Data Centre during the day of 15/7/22

This should have no impact on services but we treat this ‘at risk’ in case of any unexpected issues. The only possible impact here is that the auto changeover in case of a power failure will not function – However our UPS will be available and will cover the power until the Generator can be manually started

We will update this as the work completes

At Risk – UPS Service – Huntingdon Data Centre – 14/07/2022

Our Supplier Engineers will be performing their regular service on our UPS system at our Huntingdon Data Centre during the day of 14/7/22

This should have no impact on services but we treat this ‘at risk’ in case of any unexpected issues. NOTE that we have three resilient UPS modules and only one will be worked on at a time

We will update this as the work completes

 

Emergency Planned Work [10/07/2022]

Following the recent issue on our core switch issue – our switch vendor has asked us to urgently upgrade the firmware version on our Juniper switches to a new release they have provided us to clear a bug located through their investigations.

Due to the potential of a repeat of the previous issue we are performing the upgrade this evening to ensure there is no impact on service this coming week and to ensure the service is stable and with full resilience for the week.

In normal circumstances this would be performed out of hours / however to ensure that we have the resources available should any unexpected issues occur we are performing these upgrades through this evening meaning that IF we need to reboot / upgrade the core routers we can perform this later

The customer affect of this work will be a short period of packet loss when the affected switch reloads with the latest version of the firmware – there may be a small number of sorter periods of packet loss when other (not directly connected) switches reboot.

We anticipate the switch work will complete in the next 20 minutes

Again we apologise for the short notice on this change

 

 

Switch Issue – Telehouse West [resolved]

We are aware of an issue affecting one of our core switches in Telehouse West. this is also causing some disturbance in connections on other traffic intermittently.

Our engineers are working on this and we hope to have an update in the next 30 minutes

[resolved 01:30]
On-site engineers have rebooted one of our core switches in Telehouse West ans service is now back. Our network engineers are investigating the cause of this and we will work to ensure there is no further disruption here

Switch Upgrades – Telehouse 7/4/22 22:00 – 8/4/22 6:00

We are planning to replace our Juniper EX Series core switches in Telehouse with a newer more powerful model to allow for network growth and increased stability.

We are currently planning on working on the first two sites in Telehouse on Thursday 7th April – we will be working on one site at a time. During the work services will be at risk and access to directly connected connected servers and routers will be affected. Switches will be pre-configured and we hope that the outage per site will not exceed 2 hours. Once we are happy that all woek on one site is complete and the network is stable we will then work on the second site.

In many cases services connected will move to an alternative site – However with some connections including OFNL lines, these will be down for thr duration of the work.

We will update post before and during the work

Emergency Planned Work :: Switch Reboot in Telehouse West

We have seen a few brief drop outs during today for a small number of customers due to a switch in Telehouse West having forwarding issues on a number of core VLANs passing thought that Data Centre.

We’d been planning to announce a switch reboot late this evening to resolve this – however the drops have become more regular this afternoon and we have taken the decision to reboot the switch this afternoon to clear the issue.

We are aware of the cause of the issue (a bug in Juniper OS that sometimes strikes after  a config update and had plans to upgrade the switches to  more powerful model last week – However as one of our engineers has COVID this work has been slightly delayed.

We can see the switch is now back up and passing traffic – sorry for the lack of notice here but we wanted to restore the stability as quickly as we could and ensure that all services work as they should

Additional Broadband Gateways

Over the next week or so customers may see some additional gateways when connecting with their broadband connections.

We have had these installed and tested in our racks for a few months and are following testing we are happy these are ready for customer use. We don’t suspect any issues for customers – however if anything is seen we will just move the connections back to the original gateways.

The New gateways give us extra capacity and should add additional resilience too.

We will let you know how these go – any customers who have questions or think they may have an issue as a result please contact support via the normal routes

At Risk – Huntingdon Data Centre 24/3/2022 – 25/3/2022

We have been informed that Virgin Media will be performing a re-route on one of the backhaul links between London and our Huntingdon Data Centre between the 24/3/2022 and 25/3/2022

We have resilient links to provide service here however this will reduce the resilience for services in Huntingdon and also reduces resilience to Manchester

The planned work has a 10 hour window however we don’t anticipate the link being down that long.

We will update this post if we see any impact and once the line is restored.

At Risk – UPS Battery replacement Huntingdon – 14/3/22 [completed]

During Monday 14th March our specialist contractors will be replacing the batteries on our UPS in our Huntingdon Data Centre.

While this should have no impact on services – it does pose a slight risk and reduces resilience to services in our Huntingdon Data Centre

We will update as this work progresses

[completed 14:00]
The battery replacement is completed and the UPS and resilience are now back to normal in our Huntingdon Data Centre