[CLEAR] Telehouse North router maintenance 13th Nov, from 10pm onwards

[UDATE] All work completed to plan.

We plan on making some further changes to a couple of our core network routers to isolate and correct a service setting that is causing a few BGP issues.

This may mean a few blips in routing tables as the changes are disseminated and an occasional up-tick in latency as routes converge and stabilise but we expect no downtime or any significant service issues.

If this planning changes, we will update this ticket

RESOLVED: Outage affecting some Merula services inc. ADSL

15:15  The problem has been traced to one of our core routers which ‘hung’ without (as it should have) notifying the automatic monitoring system of problems. This in turn affected routing for some customers connected to our Telehouse data centre. Some other important switches and routers were also impacted as they were unable to see that this router was down and also failed silently.

We have now restored service to the router and are monitoring for any further issues.

This should not have happened but despite our planning it did and we can only apologise for this; we are working on reconfiguring the core layout very shortly to make sure that this can’t cause such cascading problems for our customers again.

13.30 — we believe that we have resolved these service issues. That said, we are monitoring still and looking for the root cause. Again our apologies for this loss of service and updates will continue to be posted here once we’ve had a chance to check logs etc.

UPDATE: we’re working with our link team as this is mainly affecting services out of our London data centres.  Apologies for this extended down-time, we’re all working on this problem and will update here as we know more detail.

We’re aware of and are investigating the cause of outages affecting a number of services inc. some leased lines, ethernet circuits and broadband lines. As soon as we know the root cause & likely time to fix, we’ll update there.

WARNING: fake domain registration emails

Attention: Phishing emails are circulating.

Currently, waves of forged emails are being sent to customers of several registrars, including Merula.

The subject of these emails is ‘Domain xy.com Suspension Notice’; the faked sender is shown as ‘abuse@xyzdomain’.

These emails are fakes – please ignore and delete them and do not click on any included link.

In case you have any doubts regarding emails regarding your domains, please don’t hesitate to get in touch with support in the usual ways.

[UPDATED]: Planned maintenance works on one of our transit links, London 25th/26th Nov

UPDATE: the supplier has confirmed the new date, with timings unchanged for this work.

This should not affect any services or connectivity. There may be a brief transient period when routes flap until the routing tables converge again.

Start of maintenance: 25/11/2015 – 00:00 GMT
End of maintenance: 26/11/2015 – 04:00 GMT
Duration of work: Up to 30 minutes during the window

Classification of works: Standard
Description of works: Essential Third Party maintenance

[CLOSED] ADSL issues in Suffolk & Essex

We’ve been advised by BT Openreach that since around 9:46Am today, they’ve been experiencing connectivity problems with a number of their ADSL lines especially in these two counties caused by what they believe is a broken/failed back-haul cable. There’s no time to fix given yet.

Service restored: FINAL: The onsite cable repair team completed the activities by 03:55. All Broadband services were prioritised and restored at either 00:15 or 00:30.

[UPDATE] Cable incident has now been raised as the fault has been proved out from both ends (Colchester & Ipswich). Precision testing officers (PTO) are working on localizing the fault which is 4.72km away from Ipswich telephone exchange. A PTO is in the field driving to the location of Bourne Hill which is in the Wherstead area of Ipswich. The fault has yet to be confirmed but on initial findings it looking like a 96 fibre cable that has been affected.

[COMPLETED] Planned maintenance work on transit link 20th August

We’ve been advised by Cogent Network, one of our transit suppliers, of work to be undertaken on some of their core routers. This should not mean any loss of service to us or our end-users as we have multiple diverse routes into and out of our data centres — there may be a few moments of route instability until they re-converge and stabilise.

Date and Time of Maintenance: 20/08/2015 00:00 to 04:00 GMT (01:00 to 05:00 BST)

Impact: Loss of service for up to 15 minutes during the outage window