Sep 24, 2021 | Outages
We are aware of a routing issue affecting traffic to some destinations currently. The NOC team are aware of this and are working to resolve this – we expect this to be resolved very shortly and will update as we know more
[Update 8:30]
We believe the reachability issue is now resolved – there is some further work taking place to follow up – however this will not have any impact at this time. We will work to bring the failed link back to restore redundancy shortly however this will not have any impact on services at this time.
Apr 14, 2021 | Information, Outages, Planned Work, Unplanned downtime
Once again, please accept our apologies for the problems you’ve seen over the previous couple of days. We realise that this has caused you all serious issues and for that, we’re very sorry.
Various internal changes have been implemented over the last 48 hours and currently, we believe that the network and associated services are now stable and will remain that way. We continue to monitor the situation closely to ensure that our network remains stable and there’s no further impact to your services.
Please email us in the normal way if you have any question or concerns. Thanks again for your support through this incident.
Apr 13, 2021 | Information, Outages, Unplanned downtime
23:30 Again, our apologies
We continue to undertake remedial work to mitigate this ongoing attack.
We will update here as usual.
————
We are currently seeing a new, large-scale DDoS attack against our IP range. We are working to mitigate this but some services are being affected, with packet loss, routing failures or intermittent outages. Some email delivery will be queued until this is resolved.
We will update here as usual.
13:06 UPDATE
We are mitigating a large portion of this attack traffic but currently, the transit links remain saturated which is causing the current ongoing problems. We continue to work to resolve this as quickly as possible & apologise for the ongoing inconvenience caused.
13:44 UPDATE
We are seeing most services recovering. The attack target remains offline but we believe that this incident is now contained. We apologise again for this interruption in service. If you are still seeing issues, please restart your equipment. Tickets can be raised now in the normal manner & the support line remains ready to assist.
17:43 UPDATE
The offsite server that hosts the status NOC site went down during the afternoon. Purely coincidental, but it meant we weren’t able to access it to add more frequent updates. It’s now back and we’ll update the status on the DDoS attack issues shortly. Our apologies that this wasn’t available when it was needed the most.
Apr 10, 2021 | Information, Outages, Unplanned downtime
We are currently seeing a large scale DDoS attack against our IP range.
This will lead to significant packet loss and access issues to our customers. Our NOC team are already at work to mitigate this. We will post a further update as soon as we have it
[Update 11/04/21 – 10:00am]
We believe the issue cleared shortly after 7pm yesterday. We are still monitoring this closely however we do not believe there is currently any ongoing customer impact
Mar 9, 2021 | Outages, Unplanned downtime, Update
We have seen alerts that some circuits from Virgin Media have dropped. We believe this is a Virgin Media issue but are currently investigating.
We will update this status as soon as we know more and within the hour at the latest.
UPDATE: 16:09
Virgin have confirmed an issue at one of their hub sites affecting some parts of Hertfordshire. Senior engineers and the Core Incident team are working on this at the moment.
We will have another update — if it’s not restored — within approx 30 minutes.
UPDATE: 16:20
We are seeing the affected circuits back on line – at this point we don’t have an ‘all clear’ so they should be seen at risk – but we hope this is resolved
RESOLVED 16:20
We have received the following update from Virgin Media
Please be advised that @ 16:22
Restoration Details:IOM Card 2 and MDA Card 1 remotely reseated on T0090 Luton Metnet 2a.
restoring all services.
This appears to have been a card failure at one of the Virgin Media Hub Sites – we apologise for the issues that occured here
Apr 25, 2020 | Outages
We are aware of some instability on our core London network – this has caused a number of short periods of packet loss this afternoon. Our NOC team are investigating this currently. We will update this as we know more.
This is being worked on as a high priority currently
We apologise for any issues this may cause
[update 19:10]
Investigations are ongoing on the cause of this issue. Our engineers continue to work to understand and resolve the root cause of this issue. We can see BGP sessions drop between our core routers but do not fully understand the actual cause of this. Many users may not notice the issue however there will be occasional packet loss and ‘strange’ routes while this is being worked on
[Update 20:30]
While this issue has improved this is still ongoing. We are seeing BGP between two of our core routers (Telehouse North and East) flapping. This is causing some packet loss / routing drops. now we have isolated the reason for the drops we are working to find why this is ongoing. Initial thoughts where that the routers had stale data and we therefore rebooted them – However the issue while improved has not gone away. We will update this further shortly
[Update 23:00]
The network has now been stable for over an hour and we believe the issue is cleared. we will however monitor this overnight before we close this. It appears that this was down to a forwarding issue on a core Juniper Switch in Telehouse West. After ruling out other (more likely) causes, this switch was reloaded approx 22:00 and so far the routing issues and flaps have calmed.
As the switches showed no obvious errors or log issues we will not close this until we are certain – and will investigate in more detail. However we hope there is no further customer issues as a result of this. We are sorry if this issue affected your service today
[Update 26/4/202 10:00]
The network has been stable overnight following the reboot of the core Juniper Switch. We are continuing to monitor – however we do believe this issue is now resolved.
We are sorry for any issues caused by this problem
