Planned work — transit links 12th July
Two of our supplier transit links are receiving software updates between 1am and 7am. There should be no down-time and minimal effects other than a few transient routing issues whilst traffic re-converges.
Planned Work – thn-gw2 05/06/15
We are seeing an issue with one of our Telehouse Gateway Routers. To resolve this we will need to reboot this later this evening to resolve the problem we are seeing
This should not cause an outage but may cause a routing blip as the routes transiting the router re-converge. We hope the router will not be down for more than 30 minutes – although we will will be monitoring this and will update here as needed.
[CLOSED]: Routing issues esp. affecting VOIP circuits
UPDATE 28th May: the link was re-enabled overnight and was monitored. It is now stable & we are confident that the problems it caused have now been fixed by LINX.
UPDATE: we have temporarily disabled this link and routes are now quickly recovering. We will update when this issue is fixed at LINX and we re-enable this connection.
We (and other ISPs) are seeing some instability at one of the LINX peering points today, which may cause slowness to some locations etc and strange routing paths. This is particularly affecting many VOIP circuits but other transits are also showing problems.
We are monitoring and waiting for an update from LINX – if the instability becomes too great we will disable our peering; however as other ISPs also connect there, this may not resolve all issues and we’re actively investigating alternatives.
SECURITY: possible consumer router vulnerabilities
If you have a domestic-grade router you should check by reference to this report, whether or not it may be affected by a recently notified vulnerability https://www.sec-consult.com/fxdata/seccons/prod/temedia/advisories_txt/20150519-0_KCodes_NetUSB_Kernel_Stack_Buffer_Overflow_v10.txt
If you believe this may be the case you should contact the manufacturer direct or check their web-site to see if your model has been patched and proceed accordingly.
Router Reload [10/05/2015]
Due to a memory leak one of our core routers (thn-gw2) has just rebooted. We are seeing all services back as they should be.
As this is one of a resilient pair of routers there should not have been any outage – BUT there may have been a brief ‘blip’ while the network switched over top the backup router (and another when thn-gw2 returned to service).
We are monitoring the issue and will update this post if we see any further issues
[CLOSED] Intermittent packet loss on one transit circuit
UPDATE:
Through investigations Level3 isolated the trouble to overutilization on a link from Toronto to Chicago. Traffic was rerouted to avoid the trouble and services are now restored. Level3 will continue monitoring services to ensure continued stability.
Please be advised that one of our transit providers, Level3, are experiencing an issue that may result in packet loss to some destinations. We will keep you updated as and when we receive more information. If necessary, we will disable this link until they resolve the problem
Planned work on one of our back-haul links 14th May
Between 23:00 and 05:00 on the night of 14th running into 15th, one of our back-haul providers is making some configuration changes. At some point during this window there may be a brief (30 second or so) increase in lost packets but there should be no loss of Internet connectivity. During this same brief window, the connections to any servers hosted at our Manchester POP will be at risk.
Network Blips today [04/05/2015]
We are seeing packet loss on one of the 3 links from our Huntingdon Data Centre to London. We have shut the link down which has eliminated the packet loss – BUT has left slightly less resilience on our network.
Late this evening we will be re-enabling the link and running some tests – enabling us to report the issue to our provider as needed. During the tests there may be short periods with reduced access (or packet loss) to our servers in Huntingdon
We apologise for any issues caused
UPS maintenance work 15th May
Our supplier is undertaking their regular maintenance & checking work on our main UPS at the Huntingdon data-centre on this date. There should be no impact on the day to day operations and this is informational only.
Emergency mail-server down-time today [complete]
Following the problems we’re still seeing as part of the over-running planned upgrade to our email infrastructure, we’ve decided that it’s necessary to take the mail-servers off-line at 5pm today for a period of up to 2 hours to allow this syncing to proceed quickly to completion.
Once that has occurred and run to completion, we’re confident that the new system will respond correctly as planned.
Update: The primary server is now back in service with all emails – We will be working on the backup server tomorrow – however we do not expect this to be service affecting.
Update: The mail servers are now back as a resilient pair – initial testing has shown this to be much faster and reliable – any issues please contact support