Need help or advice?
Call us FREE 0800 298 2375

Outages

Network Blip 10/07/2016

Due to a crash in one of our core routers in Telehouse North a few minutes ago this rebooted causing a very brief loss of service to come customers as the backup router took over the routing.

Customers directly connected would have seen a slightly longer outage. All routers are back up and running and full service should be restored. If you have a broadband service with us and are seeing an issue PLEASE try to reboot as this should solve any remaining issues.

We apologise if not all calls where answered during the issue- the team where busy answering calls and ensuring the issue was resolved as quickly as possible

Notification of Planned Work: 12th August Telehouse North – Virgin Media Business

We need to let you know about some maintenance work that will be taking place on our network. We will be upgrading one of our network back-haul links into the VMB node at our London, Telehouse North site.

Planned Work Start (Local UK): 12/08/2016 00:01 Planned Work End (Local UK): 12/08/2016 06:01

Impact: There will be a loss of service at some point during the timeframe stated above whilst we complete the maintenance work. You will see several interruptions to your service during the 2 hour outage window. We have planned and believe that all circuits will automatically return to service after the work is completed

Only those circuits supplied via VMB will be affected. Please email into support in the normal manner if you have any questions.

[CLOSED] OUTAGE: VOIP service

[update 17:40]

We believe that all connections have now been restored. If you still have VOIP problems, please raise a ticket in the usual manner.

We apologise for this down-time and are actively investigating alternatives and fail-over solutions as this is the second such outage in a few months.

[update: 16:00]

We understand that services are slowly being restored. This may take some time so please bear with us — we’re in the hands of the external supplier here. Further updates will follow as we get them through from the supplier network.

[update 14:52]

The supplier advises that the incident currently affecting their network is down to a Denial of service attack. They are now working with transit vendors to stop the attack and restore the network. Further updates will follow.

[update at 13:49]

Please be advised that the supplier is continuing to investigate the cause of this issue and apologise to all our customers who are affected by this. Further updates will follow.

We’re aware of issue affecting our core supplier which has dropped all of our VOIP circuits including our own PBX. There is no time to fix yet but we’ll update this ticket as we know more.

UPDATE: upgrading Virgin Media fibre into 5 Avro Court 23rd & 24th March

In preparation for additional fibre capacity being brought into the building tomorrow evening — with work commencing on both dates at 6pm — tonight we will force the traffic over our other two backup supplier back-haul channels to ensure that when the Virgin circuit is swapped over to the new fibre connections tomorrow, no down-time is seen.

This may result in some very temporary route instability and slight increase in routing times seen as the work starts & ends tonight and tomorrow. This should resolve automatically.

As always, if you have any concerns, please contact support in the usual manner.

INFORMATION: broadband connections

Tonight we are performing an urgent security update on a number of our core routers that handle FTTC and ADSL connections.

This will manifest as a brief loss of connectivity, possibly twice during the evening but only for a period of a couple of seconds as the sessions are dropped and fail-over to one of our other routers.

The connection should automatically re-establish itself. If after a power-down of your router for 10 minutes and a re-boot you still aren’t able to re-connect, please email into support@merula.net for assistance.

SECURITY PATCHES: Vulnerability Summary for CVE-2015-7547 – glibc

Since the 18th February, we have been working to ensure that all in-house systems affected by this advisory are being patched accordingly. We have been advised and judge that this work needs to be completed as quickly as possible. This may therefore require brief windows of downtime affecting both single and multiple services & circuits.

Wherever possible we will post an early notification of such customer affecting works.

If you have any concerns or require further information, contact support in the normal meaner.

Vulnerability Summary for CVE-2015-7547

Original release date: 02/18/2016
Last revised: 02/19/2016
Source: US-CERT/NIST

Overview

Multiple stack-based buffer overflows in the (1) send_dg and (2) send_vc functions in the libresolv library in the GNU C Library (aka glibc or libc6) before 2.23 allow remote attackers to cause a denial of service (crash) or possibly execute arbitrary code via a crafted DNS response that triggers a call to the getaddrinfo function with the AF_UNSPEC or AF_INET6 address family, related to performing “dual A/AAAA DNS queries” and the libnss_dns.so.2 NSS module.

Impact

CVSS Severity (version 3.0):
CVSS v3 Base Score: 8.1 High
Impact Score: 5.9
Exploitability Score: 2.2
CVSS Version 3 Metrics:
Attack Vector (AV): Network
Attack Complexity (AC): High
Privileges Required (PR): None
User Interaction (UI): None
Scope (S): Unchanged
Confidentiality (C): High
Integrity (I): High
Availability (A): High

Network Instability 02/02/2016

There was a period of network instability between 1am and 2am this morning. This appears to have been due to a network storm within our network. We have now stabalised the network and you should see full service from all servers.

We are monitoring the network to ensure this does not re-occur and are investigating the root cause of this issue to ensure it does not re-occur

We apologise for any issues this may have caused

RESOLVED: Outage affecting some Merula services inc. ADSL

15:15  The problem has been traced to one of our core routers which ‘hung’ without (as it should have) notifying the automatic monitoring system of problems. This in turn affected routing for some customers connected to our Telehouse data centre. Some other important switches and routers were also impacted as they were unable to see that this router was down and also failed silently.

We have now restored service to the router and are monitoring for any further issues.

This should not have happened but despite our planning it did and we can only apologise for this; we are working on reconfiguring the core layout very shortly to make sure that this can’t cause such cascading problems for our customers again.

13.30 — we believe that we have resolved these service issues. That said, we are monitoring still and looking for the root cause. Again our apologies for this loss of service and updates will continue to be posted here once we’ve had a chance to check logs etc.

UPDATE: we’re working with our link team as this is mainly affecting services out of our London data centres.  Apologies for this extended down-time, we’re all working on this problem and will update here as we know more detail.

We’re aware of and are investigating the cause of outages affecting a number of services inc. some leased lines, ethernet circuits and broadband lines. As soon as we know the root cause & likely time to fix, we’ll update there.

Broadband session drops

We’ve seen some sessions drop (on ADSL and FTTC/FTTP connections|) following an unexpected port reboot on one of our main switches here.

Most sessions have automatically re-connected but if your line is failing to come back as expected, it may have a stale session which needs to be cleared down. To do this, please power-down your router for 15/20 minutes and then attempt to re-connect.

This should bring you back on-line but please raise a fault with support here in the normal manner if you’re then still having problems.

Possible outage affecting a small number of C20 lines

We are aware of a problem with authentication to our Radius servers in the last 20 minutes that is affecting ONLY circuits delivered as from C20 exchanges — we are working on this now and expect a resolution shortly.

It’s only affecting those lines that have dropped earlier or had been switched off and are now trying to connect (“authenticate”) and not on lines currently logged in all of which should carry on working normally; so is only a problem for a small part of the ADSL estate in Merula.