May 11, 2023 | Information
We have had an alert of one of our fibre links between Cloud House and Telehouse is currently down, At this point it is not affecting the merula Core netwoek but does mean that we have a loss of resiliance between sites.
This is service impacting to customers taking a wave service between these two sites
The issue has been raised to the fibre provider who are actively working on the issue
We will update as we know more here
[update 11/5/23 8:00]
The fibre provider found the cause of the break and restores service on this link approx 1:45am. We have been monitoring this link since and all appears stable.
Sep 21, 2022 | Information, Planned Work
As we have previously announced due to the closure of Equinix Reynolds House in Manchester we have moved our Manchester PoP to a new location
This Friday our carrier is moving our 10G Link from Huntingdon to Manchester to the new PoP in Manchester
Through this we will be using our primary link to London so we do not expect any issues – However for a short period we will be without redundancy / backup to Huntingdon
Note in this case the work is through the daytime – as his means additional engineering resources are available to ensure the change is smooth and without issues
We will update this post as the work progresses and completes
[Completed]
Virgin Media completed the circuit move Friday evening – we now have moved all links back to our net resilient network core.
Aug 6, 2022 | Information, Planned Work
As part of our plan to enhance our network capacity and resilience over the next few days we will be migrating a number of core links to a new model of EPVN. The new config will be in place before any old links are removed. However there is a risk through this for brief drops and routing changes being visible.
This has already been tested in our lab and the configuraton is in place the final changes are just to move from the old configuration to the new setup
We apologise for the short notice for this work
We will update as the work progresses
[Update 1]
The work on the IBR core routers completed yesterday bringing our new Manchester core routers all live. There is some work this evening on our Juniper Core routers – again we don’t expect any substantal issues but you may notice some routing changes and bried periods of packet loss as the new routing converges
Jul 10, 2022 | Information, Planned Work
Following the recent issue on our core switch issue – our switch vendor has asked us to urgently upgrade the firmware version on our Juniper switches to a new release they have provided us to clear a bug located through their investigations.
Due to the potential of a repeat of the previous issue we are performing the upgrade this evening to ensure there is no impact on service this coming week and to ensure the service is stable and with full resilience for the week.
In normal circumstances this would be performed out of hours / however to ensure that we have the resources available should any unexpected issues occur we are performing these upgrades through this evening meaning that IF we need to reboot / upgrade the core routers we can perform this later
The customer affect of this work will be a short period of packet loss when the affected switch reloads with the latest version of the firmware – there may be a small number of sorter periods of packet loss when other (not directly connected) switches reboot.
We anticipate the switch work will complete in the next 20 minutes
Again we apologise for the short notice on this change
Mar 28, 2022 | Information, Planned Work
We have seen a few brief drop outs during today for a small number of customers due to a switch in Telehouse West having forwarding issues on a number of core VLANs passing thought that Data Centre.
We’d been planning to announce a switch reboot late this evening to resolve this – however the drops have become more regular this afternoon and we have taken the decision to reboot the switch this afternoon to clear the issue.
We are aware of the cause of the issue (a bug in Juniper OS that sometimes strikes after a config update and had plans to upgrade the switches to more powerful model last week – However as one of our engineers has COVID this work has been slightly delayed.
We can see the switch is now back up and passing traffic – sorry for the lack of notice here but we wanted to restore the stability as quickly as we could and ensure that all services work as they should
Mar 9, 2022 | Information
We are currently seeing an issue with some traffic routing via Telehouse West – we currently believe one of our core switches has a forwarding issue,
Our NOC team are currently working to resolve this which MAY invoice a brief reboot of the affected switch – causing a short downtime for services directly connected to that switch
We apologise for the brief and short notice reboot but want to resolve the issue as quickly as we can
[Update]
The switch is now back from the reboot and we are seeing ttraffic and routing back as it should be – we will investigate further the cause here as a background task to ensure it does not re-occur
