We are aware of some instability on our core London network – this has caused a number of short periods of packet loss this afternoon. Our NOC team are investigating this currently. We will update this as we know more.
This is being worked on as a high priority currently
We apologise for any issues this may cause
[update 19:10]
Investigations are ongoing on the cause of this issue. Our engineers continue to work to understand and resolve the root cause of this issue. We can see BGP sessions drop between our core routers but do not fully understand the actual cause of this. Many users may not notice the issue however there will be occasional packet loss and ‘strange’ routes while this is being worked on
[Update 20:30]
While this issue has improved this is still ongoing. We are seeing BGP between two of our core routers (Telehouse North and East) flapping. This is causing some packet loss / routing drops. now we have isolated the reason for the drops we are working to find why this is ongoing. Initial thoughts where that the routers had stale data and we therefore rebooted them – However the issue while improved has not gone away. We will update this further shortly
[Update 23:00]
The network has now been stable for over an hour and we believe the issue is cleared. we will however monitor this overnight before we close this. It appears that this was down to a forwarding issue on a core Juniper Switch in Telehouse West. After ruling out other (more likely) causes, this switch was reloaded approx 22:00 and so far the routing issues and flaps have calmed.
As the switches showed no obvious errors or log issues we will not close this until we are certain – and will investigate in more detail. However we hope there is no further customer issues as a result of this. We are sorry if this issue affected your service today
[Update 26/4/202 10:00]
The network has been stable overnight following the reboot of the core Juniper Switch. We are continuing to monitor – however we do believe this issue is now resolved.
We are sorry for any issues caused by this problem