Nov 16, 2015 | Information, Planned Work, Update
COMPLETED: All Work was completed during the evening on 26th November and the Data Centre now has full access to both Mains Power, Generator Power and the UPS. There was no outage to hosted servers during the work.
UPDATE: we will have engineers on site on Thursday 26th, to replace parts in our automatic transfer panel. Most of the work will not be service affecting, however there will be a short period where the data centre will be supplied by UPS alone. While we don’t anticipate any disruption, the power to our racks should be considered at risk during this period. All “at risk” work will be carried out outside core business hours and we will have staff and external electricians on site as needed to monitor/resolve any issues found.
UPDATE: the engineers are about to isolate the mains supply and generator and move us over to UPS power. This has been tested to support the data centre for at least 30 minutes but for the period of down-time, we are at risk.
We are aware of an issue with our changeover panel here and an engineer is en-route to work on this.
If the panel needs to go off-line then resilience will be reduced for a short period but the generator and UPS are still available and have just been fully tested as part of our normal weekly maintenance regime and will cut in automatically. As a reminder, the generator has fuel for at least 24-hours of continuous running.
Once the panel issue has been resolved, we’ll update this post.
Nov 13, 2015 | Information, Planned Work, Update
[UDATE] All work completed to plan.
We plan on making some further changes to a couple of our core network routers to isolate and correct a service setting that is causing a few BGP issues.
This may mean a few blips in routing tables as the changes are disseminated and an occasional up-tick in latency as routes converge and stabilise but we expect no downtime or any significant service issues.
If this planning changes, we will update this ticket
Nov 6, 2015 | Information, Planned Work
Following the outage today, we are during the course of tonight, commencing at around 10pm, making some changes to the core network routing to bring further stability & resilience online. This may mean a few blips in routing tables as the changes are disseminated and an occasional up-tick in latency as routes converge and stabilise but we expect no downtime or any significant service issues.
Nov 6, 2015 | Information, Outages, Unplanned downtime, Update
15:15 The problem has been traced to one of our core routers which ‘hung’ without (as it should have) notifying the automatic monitoring system of problems. This in turn affected routing for some customers connected to our Telehouse data centre. Some other important switches and routers were also impacted as they were unable to see that this router was down and also failed silently.
We have now restored service to the router and are monitoring for any further issues.
This should not have happened but despite our planning it did and we can only apologise for this; we are working on reconfiguring the core layout very shortly to make sure that this can’t cause such cascading problems for our customers again.
13.30 — we believe that we have resolved these service issues. That said, we are monitoring still and looking for the root cause. Again our apologies for this loss of service and updates will continue to be posted here once we’ve had a chance to check logs etc.
UPDATE: we’re working with our link team as this is mainly affecting services out of our London data centres. Apologies for this extended down-time, we’re all working on this problem and will update here as we know more detail.
We’re aware of and are investigating the cause of outages affecting a number of services inc. some leased lines, ethernet circuits and broadband lines. As soon as we know the root cause & likely time to fix, we’ll update there.
Nov 6, 2015 | Information, Update
Attention: Phishing emails are circulating.
Currently, waves of forged emails are being sent to customers of several registrars, including Merula.
The subject of these emails is ‘Domain xy.com Suspension Notice’; the faked sender is shown as ‘abuse@xyzdomain’.
These emails are fakes – please ignore and delete them and do not click on any included link.
In case you have any doubts regarding emails regarding your domains, please don’t hesitate to get in touch with support in the usual ways.
Nov 5, 2015 | Information, Planned Work, Update
UPDATE: the supplier has confirmed the new date, with timings unchanged for this work.
This should not affect any services or connectivity. There may be a brief transient period when routes flap until the routing tables converge again.
Start of maintenance: 25/11/2015 – 00:00 GMT
End of maintenance: 26/11/2015 – 04:00 GMT
Duration of work: Up to 30 minutes during the window
Classification of works: Standard
Description of works: Essential Third Party maintenance
