Aug 26, 2022 | Planned Work
Further to the work overnight the issue on our Core Harbour Exchange Switch is still not allowing config changes to be committed. At this point this is not causing any network issues – However this is obviously causing a risk and it means that we cannot provision new services on it, The errors mean there is a risk of an unexpected failure.
We have a replacement switch from the vendor which is being pre-configured during the day today
Later this evening we will swap the switch chassis from the old to the new one, This will cause some network disruption as traffic re-converges. While the switch is down services terminating in Harbour Exchange will be down and this will also affect one route into our Huntingdon Data Centre
We anticipate the outage to be approx 30-60 minutes
We are sorry for this short notice – however we wanted to ensure the issue was resolved before any customer facing issues occurred
Aug 25, 2022 | Planned Work
We have been advised of Emergency planned works affecting a number of lines this evening:
Planned Work Start (Local UK): 26/08/2022 00:01
Planned Work End (Local UK): 26/08/2022 06:00
Impact: There will be a loss of service at some point during the timeframe stated above whilst we complete the maintenance work.
The service outage should only last for up to 1 Hour, however a 6 hour time frame has been given in the event of any unforeseen circumstances.
This is for emergency planned work to repair a fault on the network in the Papworth area. Resolution for fault reference: F010061503
Customers on the lines Virgin have listed have been notified.
Aug 25, 2022 | Planned Work
Our Juniper Switch in London Harbour Exchange Square is showing an error when any new config is committed to it.
Juniper have provided their suggested work-around/fix for this and we will perform this just after midnight tonight.
There should be no impact from this work at all – However we would rather announce this as at risk in case of any unexpected impact
We will update this once the work completes
[update 26/08/22 8:30]
The work last night while not causing any issue did not resolve the issue, The vendor has supplied a replacement Core Juniper switch and this will be swapped out later tonight
Aug 21, 2022 | Planned Work
Virgin Media are performing a network upgrade on their code network between midnight and 6am. There’s an expected downtime of 5 minutes. This has been reported to affect a small number of our lines on both our London bearer or terminated in London.
From their announcement
There will be a loss of service at some point during the timeframe stated above whilst we complete the maintenance work.
The planned outage is expected to occur between 00:01am and 02:00am and should only last for up to 30 minutes or if notified for 5 minutes where this is for Network Re-convergence only. However, should we incur any card failure and a replacement is required then as a worse case we will need to use the 6 hour change window.
This work is to upgrade software on equipment on our Network at multiple sites.
We will monitor these lines and ensure that they restore during the announced window
We apologise for any issues from this work
Aug 21, 2022 | Planned Work
Virgin Media are performing a network upgrade on their code network between midnight and 6am. There’s an expected downtime of 30 minutes. This has been reported to affect a small number of our lines on both our London bearer or terminated in London.
From their announcement
There will be a loss of service at some point during the timeframe stated above whilst we complete the maintenance work.
The planned outage is expected to occur between 00:01am and 02:00am and should only last for up to 30 minutes or if notified for 5 minutes where this is for Network Re-convergence only. However, should we incur any card failure and a replacement is required then as a worse case we will need to use the 6 hour change window.
This work is to upgrade software on equipment on our Network at multiple sites.
We will monitor these lines and ensure that they restore during the announced window
We apologise for any issues from this work
Aug 6, 2022 | Information, Planned Work
As part of our plan to enhance our network capacity and resilience over the next few days we will be migrating a number of core links to a new model of EPVN. The new config will be in place before any old links are removed. However there is a risk through this for brief drops and routing changes being visible.
This has already been tested in our lab and the configuraton is in place the final changes are just to move from the old configuration to the new setup
We apologise for the short notice for this work
We will update as the work progresses
[Update 1]
The work on the IBR core routers completed yesterday bringing our new Manchester core routers all live. There is some work this evening on our Juniper Core routers – again we don’t expect any substantal issues but you may notice some routing changes and bried periods of packet loss as the new routing converges
Jul 13, 2022 | Planned Work
Our Supplier Engineers will be performing their regular service on our Generator system at our Huntingdon Data Centre during the day of 15/7/22
This should have no impact on services but we treat this ‘at risk’ in case of any unexpected issues. The only possible impact here is that the auto changeover in case of a power failure will not function – However our UPS will be available and will cover the power until the Generator can be manually started
We will update this as the work completes
Jul 13, 2022 | Planned Work
Our Supplier Engineers will be performing their regular service on our UPS system at our Huntingdon Data Centre during the day of 14/7/22
This should have no impact on services but we treat this ‘at risk’ in case of any unexpected issues. NOTE that we have three resilient UPS modules and only one will be worked on at a time
We will update this as the work completes
Jul 10, 2022 | Information, Planned Work
Following the recent issue on our core switch issue – our switch vendor has asked us to urgently upgrade the firmware version on our Juniper switches to a new release they have provided us to clear a bug located through their investigations.
Due to the potential of a repeat of the previous issue we are performing the upgrade this evening to ensure there is no impact on service this coming week and to ensure the service is stable and with full resilience for the week.
In normal circumstances this would be performed out of hours / however to ensure that we have the resources available should any unexpected issues occur we are performing these upgrades through this evening meaning that IF we need to reboot / upgrade the core routers we can perform this later
The customer affect of this work will be a short period of packet loss when the affected switch reloads with the latest version of the firmware – there may be a small number of sorter periods of packet loss when other (not directly connected) switches reboot.
We anticipate the switch work will complete in the next 20 minutes
Again we apologise for the short notice on this change
Jun 27, 2022 | Outages, Unplanned downtime
We are aware of an issue affecting one of our core switches in Telehouse West. this is also causing some disturbance in connections on other traffic intermittently.
Our engineers are working on this and we hope to have an update in the next 30 minutes
[resolved 01:30]
On-site engineers have rebooted one of our core switches in Telehouse West ans service is now back. Our network engineers are investigating the cause of this and we will work to ensure there is no further disruption here
