At Risk – Sky Leased Lines – 23/10/2024

We have been advised by Sky that they will be performing work on one of our Interconnects with them at Telehouse West on the early hours of the 23/10/24

Most Lines have dual NNI config and will swap over to the backup leg however come don’t have this so will be down during the Sky network work

We will be monitoring this work and update if we see any issues and on completion

We are sorry for any possible issues here

 

At Risk – Huntingdon & Manchester 26/10 – 27/10

We have been made aware of a Fibre repair by our provider that will affect one of the core links to our Huntingdon Data Centre from London.

The work is scheduled from 11pm on the 26th October until 2pm on the 27th. This work will not have any impact on our services as we do have resilient routes not impacted.

This does however reduce the resilience at both Manchester and Huntingdon and therefore the sites should be considered at risk.

We will be closely monitoring throughout the work and update once it completes or if there is any impact noticed

 

At Risk – Slough Data Centre [updated]

One of the two links to our Slough Data Centre is currently down – it dropped overnight.  The Data Centre has dual resiliant links so at this point service is not affected – however services at this Data Centre should be considered at risk.

The team have reported this issue to the fibre provider and we will update as we hear back

We will update this by 10am at the latest

[Update 8:10am]
Our fibre provider have acknowledged our case and have dispatched engineers to one of the Data Centres – current ETA for them to arrive is just after 10am.

[update 14:00]
Our Fibre Provider has completed OTDR testing on the fibre span and can see no fault on the fibre span – but have stated that the Data Centre Cross connect may be at fault. As such a case has been raised to have this checked. We will update on this as soon as we have it

OFNL Outage – 2/9/24 [Resolved]

We are aware of an outage affecting multiple OFNL based connections starting at approx 9:48 – we are awaiting an update from OFNL on the cause as this appears to be an issue on their network as opposed to ours.

We will update this as we know more

[Update 12:30]
We have started to see connections restore and the OFNL site itself is now back up. We have been informed that they have worked round the issue – however there may be a reduction in network capacity (and therefore some slow speeds) while work continues to fully resolve the issue

We will continue to update this post as we know more

[Resolved 14:30]
We have been monitoring the service all afternoon and can confirm that since we saw the service restore approx 12:30 we have not seen any further issues. OFNL have also confirmed the fault is cleared from their side. We & they are sorry for the issues that affected our customers today

At Rick – Huntingdon and Manchester 9/8/2024 – 10/8/2024

The supplier of one of our 10G Core links has advised us of a emergency planned works to resolve an issue with  a fibre joint in their network overnight between 10pm on the 9th August and 12 midday on 10th August

During this window one of our links between Huntingdon and Manchester will be down. Both sites have resilient links – however with reduced resilience connectivity to these locations should be considered at risk.

We will be closely monitoring our network during this window and will update this post if we see any issues

 

Loss of Resilience – Manchester and Huntingdon

We are seeing one of the core links down between Manchester and Huntingdon currently. This seems to be due to a known fault on the Virgin Media Network.

Virgin are working to resolve this issue. At this point this has no affect on our services – however there is a loss of resilience to these two sites currently as a result

We will update as we hear more

Virgin Leased Line instability [resolved]

We are seeing a small number of Virgin Media leased lines have intermittent drops – Virgin Media have located a network event. Their latest update is:

Our Network Teams have carried out investigations and they suspect the issue is due to a fibre
disturbance between Peterborough and Leicester causing the intermittent network connectivity issues.

Next steps consist of intrusive power level testing and an emergency change will be scheduled for the works activities.

Services have remained stable since 13:21 however health checks are in place to monitor services.

We’ll provide further updates as soon as we get feedback.

We are also seeing services stable but will continue to update as we have them

At Risk Planned Work – UPS Maintenance Huntingdon 5/7/2024

During the day of 5th July we will have specialist engineers working on our UPS system at our Huntingdon Data Centre.

This should not impact services at all as we have a N+1 redundant system. However there is always a slight risk and we will have slightly lower redundancy during the work

We will update this as the work progresses and completes

At Risk Planned Work – Generator Maintenance Huntingdon 22/7/2024

During the day of 22nd July we will have specialist engineers working on our Generator system at our Huntingdon Data Centre.

This should not impact services at all as as even in the case of a power failure we have a UPS system. However there is always a slight risk and we will have slightly lower redundancy during the work

We will update this as the work progresses and completes

OUTAGE – Telehouse North [resolved]

We are aware of an outage affecting our core switch in Telehouse North. This initially was a wider issue however we have now isolated this issue to services directly connected in Telehouse North.

We have an engineer en route (eta should be less than 30 minutes) and we will then have a clearer picture of the cause – and hope that service will be restored shortly thereafter

We will update this as soon as we have it

[resolved]
This issue was fully by 17:35 – One of our switches in Telehouse North had crashed – the route cause will be investigated and worked on if needed. We apologise for the issue seen here