Nov 6, 2015 | Information, Outages, Unplanned downtime, Update
15:15 The problem has been traced to one of our core routers which ‘hung’ without (as it should have) notifying the automatic monitoring system of problems. This in turn affected routing for some customers connected to our Telehouse data centre. Some other important switches and routers were also impacted as they were unable to see that this router was down and also failed silently.
We have now restored service to the router and are monitoring for any further issues.
This should not have happened but despite our planning it did and we can only apologise for this; we are working on reconfiguring the core layout very shortly to make sure that this can’t cause such cascading problems for our customers again.
13.30 — we believe that we have resolved these service issues. That said, we are monitoring still and looking for the root cause. Again our apologies for this loss of service and updates will continue to be posted here once we’ve had a chance to check logs etc.
UPDATE: we’re working with our link team as this is mainly affecting services out of our London data centres. Apologies for this extended down-time, we’re all working on this problem and will update here as we know more detail.
We’re aware of and are investigating the cause of outages affecting a number of services inc. some leased lines, ethernet circuits and broadband lines. As soon as we know the root cause & likely time to fix, we’ll update there.
Nov 6, 2015 | Information, Update
Attention: Phishing emails are circulating.
Currently, waves of forged emails are being sent to customers of several registrars, including Merula.
The subject of these emails is ‘Domain xy.com Suspension Notice’; the faked sender is shown as ‘abuse@xyzdomain’.
These emails are fakes – please ignore and delete them and do not click on any included link.
In case you have any doubts regarding emails regarding your domains, please don’t hesitate to get in touch with support in the usual ways.
Nov 5, 2015 | Information, Planned Work, Update
UPDATE: the supplier has confirmed the new date, with timings unchanged for this work.
This should not affect any services or connectivity. There may be a brief transient period when routes flap until the routing tables converge again.
Start of maintenance: 25/11/2015 – 00:00 GMT
End of maintenance: 26/11/2015 – 04:00 GMT
Duration of work: Up to 30 minutes during the window
Classification of works: Standard
Description of works: Essential Third Party maintenance
Aug 10, 2015 | Information, Update
We’ve been advised by BT Openreach that since around 9:46Am today, they’ve been experiencing connectivity problems with a number of their ADSL lines especially in these two counties caused by what they believe is a broken/failed back-haul cable. There’s no time to fix given yet.
Service restored: FINAL: The onsite cable repair team completed the activities by 03:55. All Broadband services were prioritised and restored at either 00:15 or 00:30.
[UPDATE] Cable incident has now been raised as the fault has been proved out from both ends (Colchester & Ipswich). Precision testing officers (PTO) are working on localizing the fault which is 4.72km away from Ipswich telephone exchange. A PTO is in the field driving to the location of Bourne Hill which is in the Wherstead area of Ipswich. The fault has yet to be confirmed but on initial findings it looking like a 96 fibre cable that has been affected.
Aug 8, 2015 | Information, Planned Work, Update
We’ve been advised by Cogent Network, one of our transit suppliers, of work to be undertaken on some of their core routers. This should not mean any loss of service to us or our end-users as we have multiple diverse routes into and out of our data centres — there may be a few moments of route instability until they re-converge and stabilise.
Date and Time of Maintenance: 20/08/2015 00:00 to 04:00 GMT (01:00 to 05:00 BST)
Impact: Loss of service for up to 15 minutes during the outage window
Aug 7, 2015 | Information, Outages, Unplanned downtime, Update
[Update] 15:45 The engineers have repaired the faults identified and all circuits are up & reporting as being heathy. If anyone is still seeing issues please mail into support@merula.net. We are conducting a postmortem with the supplier to ascertain the root cause, the fix made and to find out why their agreed backup routing didn’t kick in as part of the DR process. We will update the ticket as we know more.
15:10 Multiple back-hauls to us and other customers are affected by this fault; this has been escalated internally by the supplier to their 3rd-level team. No time to fix as yet.
14:26 Engineers have arrived on-site and are starting their investigations.
Engineers are en-route to both ends of this connection.
We are aware of a problem affecting one of our core bearer lines into HEX in London which in turn is affecting a number of customer leased lines. These are currently hard down.
This is a high-priority outage for us and we are working to get this resolved as quickly as possible and apologise for the down-time being seen. We do not currently have a time to fix but expect a status report within the next 30-45 minutes.
We will post updates here.
