Nov 16, 2015 | Information, Planned Work, Update
COMPLETED: All Work was completed during the evening on 26th November and the Data Centre now has full access to both Mains Power, Generator Power and the UPS. There was no outage to hosted servers during the work.
UPDATE: we will have engineers on site on Thursday 26th, to replace parts in our automatic transfer panel. Most of the work will not be service affecting, however there will be a short period where the data centre will be supplied by UPS alone. While we don’t anticipate any disruption, the power to our racks should be considered at risk during this period. All “at risk” work will be carried out outside core business hours and we will have staff and external electricians on site as needed to monitor/resolve any issues found.
UPDATE: the engineers are about to isolate the mains supply and generator and move us over to UPS power. This has been tested to support the data centre for at least 30 minutes but for the period of down-time, we are at risk.
We are aware of an issue with our changeover panel here and an engineer is en-route to work on this.
If the panel needs to go off-line then resilience will be reduced for a short period but the generator and UPS are still available and have just been fully tested as part of our normal weekly maintenance regime and will cut in automatically. As a reminder, the generator has fuel for at least 24-hours of continuous running.
Once the panel issue has been resolved, we’ll update this post.
Nov 13, 2015 | Information, Planned Work, Update
[UDATE] All work completed to plan.
We plan on making some further changes to a couple of our core network routers to isolate and correct a service setting that is causing a few BGP issues.
This may mean a few blips in routing tables as the changes are disseminated and an occasional up-tick in latency as routes converge and stabilise but we expect no downtime or any significant service issues.
If this planning changes, we will update this ticket
Nov 6, 2015 | Information, Planned Work
Following the outage today, we are during the course of tonight, commencing at around 10pm, making some changes to the core network routing to bring further stability & resilience online. This may mean a few blips in routing tables as the changes are disseminated and an occasional up-tick in latency as routes converge and stabilise but we expect no downtime or any significant service issues.
Nov 6, 2015 | Information, Outages, Unplanned downtime, Update
15:15 The problem has been traced to one of our core routers which ‘hung’ without (as it should have) notifying the automatic monitoring system of problems. This in turn affected routing for some customers connected to our Telehouse data centre. Some other important switches and routers were also impacted as they were unable to see that this router was down and also failed silently.
We have now restored service to the router and are monitoring for any further issues.
This should not have happened but despite our planning it did and we can only apologise for this; we are working on reconfiguring the core layout very shortly to make sure that this can’t cause such cascading problems for our customers again.
13.30 — we believe that we have resolved these service issues. That said, we are monitoring still and looking for the root cause. Again our apologies for this loss of service and updates will continue to be posted here once we’ve had a chance to check logs etc.
UPDATE: we’re working with our link team as this is mainly affecting services out of our London data centres. Apologies for this extended down-time, we’re all working on this problem and will update here as we know more detail.
We’re aware of and are investigating the cause of outages affecting a number of services inc. some leased lines, ethernet circuits and broadband lines. As soon as we know the root cause & likely time to fix, we’ll update there.
Nov 6, 2015 | Information, Update
Attention: Phishing emails are circulating.
Currently, waves of forged emails are being sent to customers of several registrars, including Merula.
The subject of these emails is ‘Domain xy.com Suspension Notice’; the faked sender is shown as ‘abuse@xyzdomain’.
These emails are fakes – please ignore and delete them and do not click on any included link.
In case you have any doubts regarding emails regarding your domains, please don’t hesitate to get in touch with support in the usual ways.
Nov 5, 2015 | Information, Planned Work, Update
UPDATE: the supplier has confirmed the new date, with timings unchanged for this work.
This should not affect any services or connectivity. There may be a brief transient period when routes flap until the routing tables converge again.
Start of maintenance: 25/11/2015 – 00:00 GMT
End of maintenance: 26/11/2015 – 04:00 GMT
Duration of work: Up to 30 minutes during the window
Classification of works: Standard
Description of works: Essential Third Party maintenance
Oct 9, 2015 | Information, Outages, Unplanned downtime
We’ve seen some sessions drop (on ADSL and FTTC/FTTP connections|) following an unexpected port reboot on one of our main switches here.
Most sessions have automatically re-connected but if your line is failing to come back as expected, it may have a stale session which needs to be cleared down. To do this, please power-down your router for 15/20 minutes and then attempt to re-connect.
This should bring you back on-line but please raise a fault with support here in the normal manner if you’re then still having problems.
Oct 8, 2015 | Information, Outages, Unplanned downtime
We are aware of a problem with authentication to our Radius servers in the last 20 minutes that is affecting ONLY circuits delivered as from C20 exchanges — we are working on this now and expect a resolution shortly.
It’s only affecting those lines that have dropped earlier or had been switched off and are now trying to connect (“authenticate”) and not on lines currently logged in all of which should carry on working normally; so is only a problem for a small part of the ADSL estate in Merula.
Oct 6, 2015 | Planned Work
Over the next few days we will be installing a new host link for our adsl backhaul. This should not cause any downtime but you may notice a ppp drop as sessions are balanced across the new links.
This should only cause a few seconds of issues however in some cases it may be that your router needs to be re-booted to reconnect.
If you have any issues please call support in the normal way.
Sep 9, 2015 | Information, Planned Work
We’ve been advised by Allegro, one of our transit suppliers, of work to be undertaken on one of their core routers in Manchester. This should not mean any loss of service to us or our end-users as we have multiple diverse routes into and out of our data centres — there may be a few moments of route instability during the 45 minute work window until they re-converge and stabilise.
Date of work: 17.09.15
Start time: 00:00 BST
End time: 06:00 BST
Place of work: Manchester
Planned Work: EU3819
Work description: Service migration to a new device
