One of the two links to our Slough Data Centre is currently down – it dropped overnight. The Data Centre has dual resiliant links so at this point service is not affected – however services at this Data Centre should be considered at risk.
The team have reported this issue to the fibre provider and we will update as we hear back
We will update this by 10am at the latest
[Update 8:10am]
Our fibre provider have acknowledged our case and have dispatched engineers to one of the Data Centres – current ETA for them to arrive is just after 10am.
[update 14:00]
Our Fibre Provider has completed OTDR testing on the fibre span and can see no fault on the fibre span – but have stated that the Data Centre Cross connect may be at fault. As such a case has been raised to have this checked. We will update on this as soon as we have it
We are aware of an outage affecting multiple OFNL based connections starting at approx 9:48 – we are awaiting an update from OFNL on the cause as this appears to be an issue on their network as opposed to ours.
We will update this as we know more
[Update 12:30]
We have started to see connections restore and the OFNL site itself is now back up. We have been informed that they have worked round the issue – however there may be a reduction in network capacity (and therefore some slow speeds) while work continues to fully resolve the issue
We will continue to update this post as we know more
[Resolved 14:30]
We have been monitoring the service all afternoon and can confirm that since we saw the service restore approx 12:30 we have not seen any further issues. OFNL have also confirmed the fault is cleared from their side. We & they are sorry for the issues that affected our customers today
The supplier of one of our 10G Core links has advised us of a emergency planned works to resolve an issue with a fibre joint in their network overnight between 10pm on the 9th August and 12 midday on 10th August
During this window one of our links between Huntingdon and Manchester will be down. Both sites have resilient links – however with reduced resilience connectivity to these locations should be considered at risk.
We will be closely monitoring our network during this window and will update this post if we see any issues
We are seeing one of the core links down between Manchester and Huntingdon currently. This seems to be due to a known fault on the Virgin Media Network.
Virgin are working to resolve this issue. At this point this has no affect on our services – however there is a loss of resilience to these two sites currently as a result
We will update as we hear more
We are seeing a small number of Virgin Media leased lines have intermittent drops – Virgin Media have located a network event. Their latest update is:
Our Network Teams have carried out investigations and they suspect the issue is due to a fibre
disturbance between Peterborough and Leicester causing the intermittent network connectivity issues.Next steps consist of intrusive power level testing and an emergency change will be scheduled for the works activities.
Services have remained stable since 13:21 however health checks are in place to monitor services.
We’ll provide further updates as soon as we get feedback.
We are also seeing services stable but will continue to update as we have them
During the day of 5th July we will have specialist engineers working on our UPS system at our Huntingdon Data Centre.
This should not impact services at all as we have a N+1 redundant system. However there is always a slight risk and we will have slightly lower redundancy during the work
We will update this as the work progresses and completes
During the day of 22nd July we will have specialist engineers working on our Generator system at our Huntingdon Data Centre.
This should not impact services at all as as even in the case of a power failure we have a UPS system. However there is always a slight risk and we will have slightly lower redundancy during the work
We will update this as the work progresses and completes
We are aware of an outage affecting our core switch in Telehouse North. This initially was a wider issue however we have now isolated this issue to services directly connected in Telehouse North.
We have an engineer en route (eta should be less than 30 minutes) and we will then have a clearer picture of the cause – and hope that service will be restored shortly thereafter
We will update this as soon as we have it
[resolved]
This issue was fully by 17:35 – One of our switches in Telehouse North had crashed – the route cause will be investigated and worked on if needed. We apologise for the issue seen here
We have seen one of our Virgin Media bearers drop this morning which has affected a number of our leased lines. This has been reported to Virgin who are actively investigating this.
We are sorry for any issues seen and will update this case within 1 hour
We have seen one of our core IBR routers crash this afternoon which affected reachability for a subset of customers
Service is now back however we would like to upgrade and reboot the affected devices later this evening to bring them to the latest code and reduce the risk of any further issues.
There are currently four routers left from the upgrade process mentioned in our previous post – we will aim to complete the upgrade and reboot them later this evening and if necessary tomorrow evening – though we hope to complete the work in one evening.
This should have minimal impact to customers as traffic should just route via other core routers but it is obviously ‘at risk’
[update 28/04 9AM]
All work on this completed overnight and we saw very minimal if any impact to customers. All core routers are upgraded and working as expected. If you see any issues related to this please do contact support via the normal routes