During the day of 22nd July we will have specialist engineers working on our Generator system at our Huntingdon Data Centre.
This should not impact services at all as as even in the case of a power failure we have a UPS system. However there is always a slight risk and we will have slightly lower redundancy during the work
We will update this as the work progresses and completes
We are aware of an outage affecting our core switch in Telehouse North. This initially was a wider issue however we have now isolated this issue to services directly connected in Telehouse North.
We have an engineer en route (eta should be less than 30 minutes) and we will then have a clearer picture of the cause – and hope that service will be restored shortly thereafter
We will update this as soon as we have it
[resolved]
This issue was fully by 17:35 – One of our switches in Telehouse North had crashed – the route cause will be investigated and worked on if needed. We apologise for the issue seen here
We have seen one of our Virgin Media bearers drop this morning which has affected a number of our leased lines. This has been reported to Virgin who are actively investigating this.
We are sorry for any issues seen and will update this case within 1 hour
We have seen one of our core IBR routers crash this afternoon which affected reachability for a subset of customers
Service is now back however we would like to upgrade and reboot the affected devices later this evening to bring them to the latest code and reduce the risk of any further issues.
There are currently four routers left from the upgrade process mentioned in our previous post – we will aim to complete the upgrade and reboot them later this evening and if necessary tomorrow evening – though we hope to complete the work in one evening.
This should have minimal impact to customers as traffic should just route via other core routers but it is obviously ‘at risk’
[update 28/04 9AM]
All work on this completed overnight and we saw very minimal if any impact to customers. All core routers are upgraded and working as expected. If you see any issues related to this please do contact support via the normal routes
We are planning to start a rolling project of upgrading our core routers over the next few weeks – this will involve the LNS routers (those that Broadband customers connect to) and our Internet facing border routers.
In the case of the LNS / Broadband routers this will lead to a brief drop and re-connect of your service when the router is rebooted after the update.
In the case of the core routers this should not be visible as in most cases traffic will simply re-route round them and we will remove traffic from the router we are working on before upgrading it
We will update this posy shortly with a schedule and update on the project
[completed]
All upgrades have been completed for this work.
We are seeing one of our Virgin Media NNIs down currently affecting services that are provided on the affected NNI
This is being raised with Virgin Media currently and we will update this post as we have mode details.
NOTE this only affects leased lines on the one affected link – other services are working.
[Update 13:10]
We have the following update from Virgin Media
We’re aware of an issue impacting your servicesNetwork Teams have identified a number of 10gb High Capacity Services have lost connectivity.An Engineer had already been requested and arrived onsite in Northampton at 12:13 and investigations are underway. All respective support teams have been engaged to progress investigations and create a fix plan.
We will update as we know more
[Update 14:00]
We have seen the affected NNI return to service just before 2pm. We do not have an update from Virgin to confirm that this is fully resolved – but for now we see services back to normal. We do treak them as ‘at risk’ until the issue is closed down.
We have been advised by Virgin Media they will be performing work on a backup link between our Huntingdon and Manchester Data Centre.
The work is scheduled for the early hours of the 19th March.
We do not anticipate any downtime from this but the network does have reduced resilience while this work is being undertake,
Due to a warning on one of our Broadband Routers in Telehouse West we may been to upgrade this and reboot it later this evening.
If your session is affected your session is affected you should fine you re-connect within a few minutes
If for any reason the service continues to be down – please then reboot your router
We will update this after the work completes – Sorry for any issues
We are seeing a minor error on one of our switches in HEX for the past few weeks. We have tried a number of non service affecting fixes – which have unfortunately not cleared the issue.
The warnings at this point are not service affecting – however we want to clear this before it becomes a issue.
We are in the data centre for some other work tomorrow – and if time allows we will run some checks while we have an engineer there.
There is a risk we may need to power cycle the switch to try and clear the issue. Depending on what we find we may perform this tomorrow or we may schedule this another time.
There is no core routing via this switch however there are links to some leased line and broadband connections on the switch – IF the switch is rebooted most of these will see a brief drop while the sessions move to alternative links
We will update this as we know more
[update]
We performed some initial work here but will re-plan the re-boot / replacement for another date.
One of our wholesale partners are upgrading the switch that one of our feeds is connected to over night on the 21st November
We will gracefully move sessions to the second NNI before the work starts and re-enable the affected link after their work.
Customers with sessions on the affected link may see their connection drop briefly when this work takes place