Network issue this morning 27/05/2013

We are aware of a network issue that occurred this morning. This appears to have been caused by one of our routers crashing.

We do have multiple routers so normally this would not cause a problem – however in this case it would appear that while the router stopped routing some traffic properly it was still advertising the route to our other routers.

We have now reset the failed router which so far seems stable – we are monitoring the router for any signs of issues

We are sorry for any problems caused by this issue.

Webserver issue [resolved]

One of our shared hosting web servers (cgi.merula.net / 217.146.105.1) had some issues earlier today due to a attack against a number of scripts on the server.

We believe that the attack has now been blocked and the issue resolved.

We are sorry for any issues this may have caused

Post.merula.co.uk outage -RESOLVED

Or primary outgoing smtp server crashed earlier today due to a memory leak.

The server was rebooted which restored service. We are investigating the root cause to see if we can prevent this issue re-occurring.

We are sorry for any issues caused

OUTAGE: 10 minute outage today [resolved]

We apologise for this brief window of down-time caused by a misconfiguration on a third party suppliers link. This meant that a network loop was introduced into our network today causing a brief outage for some users.

The core switch here recovered automatically after about 10 minutes and took the link out of action and we have left it down while we investigate the cause in more detail.

We will post more as soon as we know.

FINAL UPDATE: ADSL outages

UPDATE3:

BT have finally reported that this has been cleared. We still have a few users not logged in correctly. The fix for this is either a router re-boot and leave it switched off for at least 30 minutes, or, if that’s been tried and still fails to establish a connection, we can arrange for a line reset. If you’re in this position, please drop an email to support@merula.net or call in on the helpline and we’ll get that working for you.

We’ll advise here when BT confirm exactly what went wrong. The emergency work was completed around 5pm and this had caused overnight instability. At 5.30, the Stepney Green system went down again. And stayed like that until a few minutes ago.

Again, our apologies if you were affected.

UPDATE2: BT have confirmed that this is still an outstanding loss of service which they’re still working on. The last update was at 9.30 with no time to fix agreed then. We apologise to those customers still affected. Please keep revisiting this page for updates.

UPDATE: BT undertook emergency work in the Stepney Green area which should have
been completed by 4am. This is however still causing issues with
authentication. We’re waiting for a definitive answer from them on “time
to fix”.

A router re-boot has worked for some people but not all.

We’re aware of reports of outages for ADSL users both in the Merula network and elsewhere. We are investigating and will report back.

ADSL gateway restart

We are in the process of updating one of the gateway ADSL routers as we’ve been made aware of an urgent security issue that requires new firmware. Current connections are being made & authenticated against the backup router. We now have to re-boot the one under maintenance and this will mean a brief window where users may see an outage of around a second. Any new sessions will not be affected.