Jul 12, 2015 | Information, Planned Work, Update
Change of date:
We’ve been advised by Cogent Network, one of our transit suppliers of work to be undertaken on some of their core routers. This should not mean any loss of service to us or our end-users as we have multiple diverse routes into and out of our data centres — there may be a few moments of route instability until they re-converge and stabilise.
Maintenance window:
Date of work: 8.8.2015
Start time: 00:00 BST
End time: 02:30 BST
Place of work: London Telehouse North
Work description: Hardware upgrade on Core router controller cards
Jul 7, 2015 | Information, Outages, Unplanned downtime
We just saw a temporary blip on one of our interconnects which may have caused a brief < 1 minute loss of connectivity to some services hosted here in the Huntingdon data centre.
This appears to be a momentary issue only but we are investigating the root cause.
Jul 3, 2015 | Information, Planned Work
Two of our supplier transit links are receiving software updates between 1am and 7am. There should be no down-time and minimal effects other than a few transient routing issues whilst traffic re-converges.
Jun 5, 2015 | Information, Planned Work
We are seeing an issue with one of our Telehouse Gateway Routers. To resolve this we will need to reboot this later this evening to resolve the problem we are seeing
This should not cause an outage but may cause a routing blip as the routes transiting the router re-converge. We hope the router will not be down for more than 30 minutes – although we will will be monitoring this and will update here as needed.
May 27, 2015 | Information, Outages, Update
UPDATE 28th May: the link was re-enabled overnight and was monitored. It is now stable & we are confident that the problems it caused have now been fixed by LINX.
UPDATE: we have temporarily disabled this link and routes are now quickly recovering. We will update when this issue is fixed at LINX and we re-enable this connection.
We (and other ISPs) are seeing some instability at one of the LINX peering points today, which may cause slowness to some locations etc and strange routing paths. This is particularly affecting many VOIP circuits but other transits are also showing problems.
We are monitoring and waiting for an update from LINX – if the instability becomes too great we will disable our peering; however as other ISPs also connect there, this may not resolve all issues and we’re actively investigating alternatives.
May 20, 2015 | Information
If you have a domestic-grade router you should check by reference to this report, whether or not it may be affected by a recently notified vulnerability https://www.sec-consult.com/fxdata/seccons/prod/temedia/advisories_txt/20150519-0_KCodes_NetUSB_Kernel_Stack_Buffer_Overflow_v10.txt
If you believe this may be the case you should contact the manufacturer direct or check their web-site to see if your model has been patched and proceed accordingly.
May 10, 2015 | Information
Due to a memory leak one of our core routers (thn-gw2) has just rebooted. We are seeing all services back as they should be.
As this is one of a resilient pair of routers there should not have been any outage – BUT there may have been a brief ‘blip’ while the network switched over top the backup router (and another when thn-gw2 returned to service).
We are monitoring the issue and will update this post if we see any further issues
May 6, 2015 | Information, Update
UPDATE:
Through investigations Level3 isolated the trouble to overutilization on a link from Toronto to Chicago. Traffic was rerouted to avoid the trouble and services are now restored. Level3 will continue monitoring services to ensure continued stability.
Please be advised that one of our transit providers, Level3, are experiencing an issue that may result in packet loss to some destinations. We will keep you updated as and when we receive more information. If necessary, we will disable this link until they resolve the problem
May 5, 2015 | Information, Planned Work
Between 23:00 and 05:00 on the night of 14th running into 15th, one of our back-haul providers is making some configuration changes. At some point during this window there may be a brief (30 second or so) increase in lost packets but there should be no loss of Internet connectivity. During this same brief window, the connections to any servers hosted at our Manchester POP will be at risk.
May 4, 2015 | Uncategorized
We are seeing packet loss on one of the 3 links from our Huntingdon Data Centre to London. We have shut the link down which has eliminated the packet loss – BUT has left slightly less resilience on our network.
Late this evening we will be re-enabling the link and running some tests – enabling us to report the issue to our provider as needed. During the tests there may be short periods with reduced access (or packet loss) to our servers in Huntingdon
We apologise for any issues caused
