Need help or advice?
Call us FREE 0800 298 2375

Unplanned downtime

[CLOSED] OUTAGE: VOIP service

[update 17:40]

We believe that all connections have now been restored. If you still have VOIP problems, please raise a ticket in the usual manner.

We apologise for this down-time and are actively investigating alternatives and fail-over solutions as this is the second such outage in a few months.

[update: 16:00]

We understand that services are slowly being restored. This may take some time so please bear with us — we’re in the hands of the external supplier here. Further updates will follow as we get them through from the supplier network.

[update 14:52]

The supplier advises that the incident currently affecting their network is down to a Denial of service attack. They are now working with transit vendors to stop the attack and restore the network. Further updates will follow.

[update at 13:49]

Please be advised that the supplier is continuing to investigate the cause of this issue and apologise to all our customers who are affected by this. Further updates will follow.

We’re aware of issue affecting our core supplier which has dropped all of our VOIP circuits including our own PBX. There is no time to fix yet but we’ll update this ticket as we know more.

RESOLVED: Outage affecting some Merula services inc. ADSL

15:15  The problem has been traced to one of our core routers which ‘hung’ without (as it should have) notifying the automatic monitoring system of problems. This in turn affected routing for some customers connected to our Telehouse data centre. Some other important switches and routers were also impacted as they were unable to see that this router was down and also failed silently.

We have now restored service to the router and are monitoring for any further issues.

This should not have happened but despite our planning it did and we can only apologise for this; we are working on reconfiguring the core layout very shortly to make sure that this can’t cause such cascading problems for our customers again.

13.30 — we believe that we have resolved these service issues. That said, we are monitoring still and looking for the root cause. Again our apologies for this loss of service and updates will continue to be posted here once we’ve had a chance to check logs etc.

UPDATE: we’re working with our link team as this is mainly affecting services out of our London data centres.  Apologies for this extended down-time, we’re all working on this problem and will update here as we know more detail.

We’re aware of and are investigating the cause of outages affecting a number of services inc. some leased lines, ethernet circuits and broadband lines. As soon as we know the root cause & likely time to fix, we’ll update there.

Broadband session drops

We’ve seen some sessions drop (on ADSL and FTTC/FTTP connections|) following an unexpected port reboot on one of our main switches here.

Most sessions have automatically re-connected but if your line is failing to come back as expected, it may have a stale session which needs to be cleared down. To do this, please power-down your router for 15/20 minutes and then attempt to re-connect.

This should bring you back on-line but please raise a fault with support here in the normal manner if you’re then still having problems.

Possible outage affecting a small number of C20 lines

We are aware of a problem with authentication to our Radius servers in the last 20 minutes that is affecting ONLY circuits delivered as from C20 exchanges — we are working on this now and expect a resolution shortly.

It’s only affecting those lines that have dropped earlier or had been switched off and are now trying to connect (“authenticate”) and not on lines currently logged in all of which should carry on working normally; so is only a problem for a small part of the ADSL estate in Merula.

[Update] URGENT: Leased line outages

 

[Update] 15:45 The engineers have repaired the faults identified and all circuits are up & reporting as being heathy. If anyone is still seeing issues please mail into support@merula.net. We are conducting a postmortem with the supplier to ascertain the root cause, the fix made and to find out why their agreed backup routing didn’t kick in as part of the DR process. We will update the ticket as we know more.

15:10 Multiple back-hauls to us and other customers are affected by this fault; this has been escalated internally by the supplier to their 3rd-level team. No time to fix as yet.

14:26 Engineers have arrived on-site and are starting their investigations.

Engineers are en-route to both ends of this connection.

We are aware of a problem affecting one of our core bearer lines into HEX in London which in turn is affecting a number of customer  leased lines. These are currently hard down.

This is a high-priority outage for us and we are working to get this resolved as quickly as possible and apologise for the down-time being seen. We do not currently have a time to fix but expect a status report within the next 30-45 minutes.

We will post updates here.

Virtual Server – failure

One of our Virtual servers currently has a failed RAID. The drive is rebuilding with a replacement drive.

Sites currently hosted on this server are currently unavailable – this includes one of our primary name servers (which should have no affect) and a couple of customer servers along with one of our shared web sites.

We are working to bring this back as soon as possible and will update this as the disk re-build progresses. We believe at this time that no data has been lost

This does not impact any services UNLESS your site was hosted on the affected server.

RADIUS server issues affecting ADSL logins

For a brief period, the RADIUS server here in Huntingdon that authenticates ADSL logins was refusing connections. This is now resolved and any ADSL circuits that weren’t able to login should have automatically come back on line again. If you are still seeing issues, please re-start your router.

Brief network outage at approx. 15:10 7th July

We just saw a temporary blip on one of our interconnects which may have caused a brief < 1 minute loss of connectivity to some services hosted here in the Huntingdon data centre.

This appears to be a momentary issue only but we are investigating the root cause.

Some ADSL sessions may have just dropped

We had to reboot one of our core ADSL routers as we were aware of various problems affecting a significant number of lines and we took the decision that a very short outage was needed to allow these sessions to be cleared down. We have seen all of the affected lines come back successfully, so believe that this is now resloved. Apologies to anyone who briefly lost their connection.

Issues with virtual server & hosted sites

UPDATE: the fault has been isolated and fixed. Any customer facing services affected should now be back to normal.

We’ve a hardware issue that is affecting some hosted WP sites and may impact other sites also hosted on the same server. No time-to-fix information as yet; we’ll update this post as more information becomes available.

EMERGENCY CONTACTS

The main support number is 0845 330 0666 (geographical 01480 355566)

There’s a second, fallback (geographical) number: 01480 411616. All numbers ring directly at our support centre, manned 24hrs x 365 days a year.

We'd also suggest that all customers subscribe to our mailing list (link above); status messages and updates will be delivered by email.

Subscribe