At Risk – Core Link down – Loss of Resilience 15/11/2023 [resolved]

Our provider for one of our core links between London and Manchester has a fibre break meaning that our core link between London and Manchester is currently down.

This is not currently causing any issue but it does mean that we have a lack of resilience to both the Manchester and Huntingdon sites.

Our supplier have identified the fibre break location caused by a digger and are working to repair the damage currently.

We will update this as we know more

[resolved[

The fibre damage was resolved yesterday and the link has remained stable since

 

At Risk – Huntingdon Data Centre 28/1/2022 – 29/1/2022 [completed]

We have been informed that Virgin Media will be performing a re-route on one of the backhaul links between London and our Huntingdon Data Centre between the 28/1/2022 and 29/1/2022

We have resilient links to provide service here however this will reduce the resilience for services in Huntingdon and also reduces resilience to Manchester

The planned work has a 10 hour window however we don’t anticipate the link being down that long.

We will update this post if we see any impact and once the line is restored.

Note this is a re-schedule of the earlier announced work

[Update 29/1/2022 9:30]

We saw the link drop approx 12:30am and returned approx 5:30. Failover went as planned and there was no noticeable affect on traffic

 

Connectivity Issue – 11/1/2022

We saw a drop in most traffic to transit providers between 19:15 and 19:40 this evening.

This showed  as a drop in traffic over that period for any traffic via Transit (connectivity reached over our own peering was fine.The effect would have been the inability to reach certain sites dueing this time.

The issue resolved itself while we started investigating. We believe this issue was caused by an upstream network as opposed to something inside our own network

We are closely monitoring and investigating this issue and working with our upstream providers. As and when we hear back we will update this post

Earlier packet Loss [19/10/2021]

We are aware of an issue that caused a period of packet loss and potentially slow connecticity for services located or Routed via Telehouse East earlier today.

The network is now stable and not showing any issues. We are working with the Data Centre and the NOC team to fully understand the trigger for this period of instability. IF any work is needed following on from this we will update this ticket.

 

Phone Issues

Due to an on-going incident with one of our SIP providers we are aware that calls to some of our own numbers may have call quality issues, The supplier is working to resolve this. In the mean time if you wish to call us please feel free to call us on 0330 341 6666 for support or  0330 341 0000 for our main number which uses an alternative carrier.

This will still reach the same teams here to help with any query

 

Network Issue [26/09/2021] – RESOLVED

We are aware of a network issue that is causing connectivity issues for some customers currently – the effect depends which PoP your connection is on.

Our engineers are aware and have been working to resolve this urgently. We expect to have an update in the next 30 minutes

[update 2:30pm]
We believe the vast majority of the packet loss and issues seen has now cleared.We are aware of a few sites that can’t be reached and there is ongoing work – with the potential for planned work later today. If you do have a site you can’t access etc please do email support or call in since the extra information will ensure that we check and clear all remaining issues.

[update 9:30pm]
Work on the network has now finished for tonight. The network has remained stable most of the day with the exception on a small number of sites which showed issues. Troubleshooting has been performed this evening and passed to the router vendor to help locate the underlying issue. However we do understand the trigger and have worked round these for now. If you are still seeing any issues please let us know. We will close this issue shortly and raised a planned work as needed in due course,

This issue is now fully resolved and out network is now live on our new network core