We are seeing one of the core links down between Manchester and Huntingdon currently. This seems to be due to a known fault on the Virgin Media Network.
Virgin are working to resolve this issue. At this point this has no affect on our services – however there is a loss of resilience to these two sites currently as a result
We will update as we hear more
We are seeing a small number of Virgin Media leased lines have intermittent drops – Virgin Media have located a network event. Their latest update is:
Our Network Teams have carried out investigations and they suspect the issue is due to a fibre
disturbance between Peterborough and Leicester causing the intermittent network connectivity issues.Next steps consist of intrusive power level testing and an emergency change will be scheduled for the works activities.
Services have remained stable since 13:21 however health checks are in place to monitor services.
We’ll provide further updates as soon as we get feedback.
We are also seeing services stable but will continue to update as we have them
During the day of 5th July we will have specialist engineers working on our UPS system at our Huntingdon Data Centre.
This should not impact services at all as we have a N+1 redundant system. However there is always a slight risk and we will have slightly lower redundancy during the work
We will update this as the work progresses and completes
During the day of 22nd July we will have specialist engineers working on our Generator system at our Huntingdon Data Centre.
This should not impact services at all as as even in the case of a power failure we have a UPS system. However there is always a slight risk and we will have slightly lower redundancy during the work
We will update this as the work progresses and completes
We are aware of an outage affecting our core switch in Telehouse North. This initially was a wider issue however we have now isolated this issue to services directly connected in Telehouse North.
We have an engineer en route (eta should be less than 30 minutes) and we will then have a clearer picture of the cause – and hope that service will be restored shortly thereafter
We will update this as soon as we have it
[resolved]
This issue was fully by 17:35 – One of our switches in Telehouse North had crashed – the route cause will be investigated and worked on if needed. We apologise for the issue seen here
We have seen one of our Virgin Media bearers drop this morning which has affected a number of our leased lines. This has been reported to Virgin who are actively investigating this.
We are sorry for any issues seen and will update this case within 1 hour
We have seen one of our core IBR routers crash this afternoon which affected reachability for a subset of customers
Service is now back however we would like to upgrade and reboot the affected devices later this evening to bring them to the latest code and reduce the risk of any further issues.
There are currently four routers left from the upgrade process mentioned in our previous post – we will aim to complete the upgrade and reboot them later this evening and if necessary tomorrow evening – though we hope to complete the work in one evening.
This should have minimal impact to customers as traffic should just route via other core routers but it is obviously ‘at risk’
[update 28/04 9AM]
All work on this completed overnight and we saw very minimal if any impact to customers. All core routers are upgraded and working as expected. If you see any issues related to this please do contact support via the normal routes
We are planning to start a rolling project of upgrading our core routers over the next few weeks – this will involve the LNS routers (those that Broadband customers connect to) and our Internet facing border routers.
In the case of the LNS / Broadband routers this will lead to a brief drop and re-connect of your service when the router is rebooted after the update.
In the case of the core routers this should not be visible as in most cases traffic will simply re-route round them and we will remove traffic from the router we are working on before upgrading it
We will update this posy shortly with a schedule and update on the project
[completed]
All upgrades have been completed for this work.
We are seeing one of our Virgin Media NNIs down currently affecting services that are provided on the affected NNI
This is being raised with Virgin Media currently and we will update this post as we have mode details.
NOTE this only affects leased lines on the one affected link – other services are working.
[Update 13:10]
We have the following update from Virgin Media
We’re aware of an issue impacting your servicesNetwork Teams have identified a number of 10gb High Capacity Services have lost connectivity.An Engineer had already been requested and arrived onsite in Northampton at 12:13 and investigations are underway. All respective support teams have been engaged to progress investigations and create a fix plan.
We will update as we know more
[Update 14:00]
We have seen the affected NNI return to service just before 2pm. We do not have an update from Virgin to confirm that this is fully resolved – but for now we see services back to normal. We do treak them as ‘at risk’ until the issue is closed down.
We have been advised by Virgin Media they will be performing work on a backup link between our Huntingdon and Manchester Data Centre.
The work is scheduled for the early hours of the 19th March.
We do not anticipate any downtime from this but the network does have reduced resilience while this work is being undertake,