Following the outage today, we are during the course of tonight, commencing at around 10pm, making some changes to the core network routing to bring further stability & resilience online. This may mean a few blips in routing tables as the changes are disseminated and an occasional up-tick in latency as routes converge and stabilise but we expect no downtime or any significant service issues.
15:15 The problem has been traced to one of our core routers which ‘hung’ without (as it should have) notifying the automatic monitoring system of problems. This in turn affected routing for some customers connected to our Telehouse data centre. Some other important switches and routers were also impacted as they were unable to see that this router was down and also failed silently.
We have now restored service to the router and are monitoring for any further issues.
This should not have happened but despite our planning it did and we can only apologise for this; we are working on reconfiguring the core layout very shortly to make sure that this can’t cause such cascading problems for our customers again.
13.30 — we believe that we have resolved these service issues. That said, we are monitoring still and looking for the root cause. Again our apologies for this loss of service and updates will continue to be posted here once we’ve had a chance to check logs etc.
UPDATE: we’re working with our link team as this is mainly affecting services out of our London data centres. Apologies for this extended down-time, we’re all working on this problem and will update here as we know more detail.
We’re aware of and are investigating the cause of outages affecting a number of services inc. some leased lines, ethernet circuits and broadband lines. As soon as we know the root cause & likely time to fix, we’ll update there.
Currently, waves of forged emails are being sent to customers of several registrars, including Merula.
The subject of these emails is ‘Domain xy.com Suspension Notice’; the faked sender is shown as ‘abuse@xyzdomain’.
These emails are fakes – please ignore and delete them and do not click on any included link.
In case you have any doubts regarding emails regarding your domains, please don’t hesitate to get in touch with support in the usual ways.
UPDATE: the supplier has confirmed the new date, with timings unchanged for this work.
This should not affect any services or connectivity. There may be a brief transient period when routes flap until the routing tables converge again.
Start of maintenance: 25/11/2015 – 00:00 GMT
End of maintenance: 26/11/2015 – 04:00 GMT
Duration of work: Up to 30 minutes during the window
Classification of works: Standard
Description of works: Essential Third Party maintenance
We’ve seen some sessions drop (on ADSL and FTTC/FTTP connections|) following an unexpected port reboot on one of our main switches here.
Most sessions have automatically re-connected but if your line is failing to come back as expected, it may have a stale session which needs to be cleared down. To do this, please power-down your router for 15/20 minutes and then attempt to re-connect.
This should bring you back on-line but please raise a fault with support here in the normal manner if you’re then still having problems.
We are aware of a problem with authentication to our Radius servers in the last 20 minutes that is affecting ONLY circuits delivered as from C20 exchanges — we are working on this now and expect a resolution shortly.
It’s only affecting those lines that have dropped earlier or had been switched off and are now trying to connect (“authenticate”) and not on lines currently logged in all of which should carry on working normally; so is only a problem for a small part of the ADSL estate in Merula.
Over the next few days we will be installing a new host link for our adsl backhaul. This should not cause any downtime but you may notice a ppp drop as sessions are balanced across the new links.
This should only cause a few seconds of issues however in some cases it may be that your router needs to be re-booted to reconnect.
If you have any issues please call support in the normal way.
We’ve been advised by Allegro, one of our transit suppliers, of work to be undertaken on one of their core routers in Manchester. This should not mean any loss of service to us or our end-users as we have multiple diverse routes into and out of our data centres — there may be a few moments of route instability during the 45 minute work window until they re-converge and stabilise.
Date of work: 17.09.15
Start time: 00:00 BST
End time: 06:00 BST
Place of work: Manchester
Planned Work: EU3819
Work description: Service migration to a new device
One of the smaller UPS units protecting one of the racks here in the data centre is indicating that the bateries need replacing. These are on order and the work will be undertaken shortly. We will advise further when we have a confirmed date & time.
There should be no impact on the rack up-time as the main data-centre UPS will be on-line at all times, but as this rack contains a number of our core switches and there’s always the possibility of an unforseen problem, we consider that this should be noted as “at risk” work.
We’ve been advised by Allegro, one of our transit suppliers, of work to be undertaken on some of their core routers. This should not mean any loss of service to us or our end-users as we have multiple diverse routes into and out of our data centres — there may be a few moments of route instability until they re-converge and stabilise.
Date and Time of Maintenance: 02-Sep-2015 23:59:00 GMT to 03-Sep-2015 03:00:00 BST
Impact: Loss of service for up to 15 minutes during the outage window