app approx 5am this morning our main outbound email server crashed do to a memory issue.
This has been resolved and all services should have been running again just after 8:30am
We are sorry of this issue affected you in any way
Some of our users may have noticed some periods of packet loss and inability to reach certain locations today.
We have been working on this issue since the early hours of this morning – and after a lot of debugging e have traced this issue to a change made by a supplier to two of our inter-site fibre links.
We are liaising with the supplier to get this fixed and have made some temporary changes to stop the issue re-occurring however this had left the network with less automatic resilience than normal.
There is a small risk of a network break on our other services causing some areas of our network to become unreachable – however if this happens we will manually switch over to the backup links. In addition once or supplier completes work here there will be an at risk period when we swap back to all current links.
Virgin Media have now replaced a faulty NTE in our Huntingdon Data Centre which appears to have now resolved this issue. All of our links and resilient paths are now fully enabled. We will be monitoring this issue t ensure it does not re-occur.
We are sorry for any issues caused by this problem
We are aware of some issues on the LINX peering network. While this is not affecting our network at present some sites and locations may be slow as a side effect of this. We are awaiting more details from LINX and will update this as more is known
We are aware of reports of a bug affecting Juniper core routers affecting some ISPs and large interconnects that are causing these routers to crash. Symptoms include increased latency and slow speeds when attempting to load websites. Please be aware of this as it may affect your own connections if any of the intermediate links depend on these ISPs or routers.
We apologise for this brief window of down-time caused by a misconfiguration on a third party suppliers link. This meant that a network loop was introduced into our network today causing a brief outage for some users.
The core switch here recovered automatically after about 10 minutes and took the link out of action and we have left it down while we investigate the cause in more detail.
We will post more as soon as we know.
This may affect some customers but the router should re-connect and authenticate after the work is completed:
This Pew PW177211 is an emergency pew that will affect WMBC shared end users who are connected to the Leeds 21c POP.
The impact will be that end users at Leeds will experience down time between 02:00 and up to 06:00 .
The CP’s whose end users are on this site will see authentication requests but the end users will be unable to connect to their WBMC shared CP.