Feb 20, 2016 | Information, Outages, Planned Work
Since the 18th February, we have been working to ensure that all in-house systems affected by this advisory are being patched accordingly. We have been advised and judge that this work needs to be completed as quickly as possible. This may therefore require brief windows of downtime affecting both single and multiple services & circuits.
Wherever possible we will post an early notification of such customer affecting works.
If you have any concerns or require further information, contact support in the normal meaner.
Vulnerability Summary for CVE-2015-7547
Original release date: 02/18/2016
Last revised: 02/19/2016
Source: US-CERT/NIST
Overview
Multiple stack-based buffer overflows in the (1) send_dg and (2) send_vc functions in the libresolv library in the GNU C Library (aka glibc or libc6) before 2.23 allow remote attackers to cause a denial of service (crash) or possibly execute arbitrary code via a crafted DNS response that triggers a call to the getaddrinfo function with the AF_UNSPEC or AF_INET6 address family, related to performing “dual A/AAAA DNS queries” and the libnss_dns.so.2 NSS module.
Impact
CVSS Severity (version 3.0):
CVSS v3 Base Score: 8.1 High
Impact Score: 5.9
Exploitability Score: 2.2
CVSS Version 3 Metrics:
Attack Vector (AV): Network
Attack Complexity (AC): High
Privileges Required (PR): None
User Interaction (UI): None
Scope (S): Unchanged
Confidentiality (C): High
Integrity (I): High
Availability (A): High
Feb 2, 2016 | Outages
There was a period of network instability between 1am and 2am this morning. This appears to have been due to a network storm within our network. We have now stabalised the network and you should see full service from all servers.
We are monitoring the network to ensure this does not re-occur and are investigating the root cause of this issue to ensure it does not re-occur
We apologise for any issues this may have caused
Nov 6, 2015 | Information, Outages, Unplanned downtime, Update
15:15 The problem has been traced to one of our core routers which ‘hung’ without (as it should have) notifying the automatic monitoring system of problems. This in turn affected routing for some customers connected to our Telehouse data centre. Some other important switches and routers were also impacted as they were unable to see that this router was down and also failed silently.
We have now restored service to the router and are monitoring for any further issues.
This should not have happened but despite our planning it did and we can only apologise for this; we are working on reconfiguring the core layout very shortly to make sure that this can’t cause such cascading problems for our customers again.
13.30 — we believe that we have resolved these service issues. That said, we are monitoring still and looking for the root cause. Again our apologies for this loss of service and updates will continue to be posted here once we’ve had a chance to check logs etc.
UPDATE: we’re working with our link team as this is mainly affecting services out of our London data centres. Apologies for this extended down-time, we’re all working on this problem and will update here as we know more detail.
We’re aware of and are investigating the cause of outages affecting a number of services inc. some leased lines, ethernet circuits and broadband lines. As soon as we know the root cause & likely time to fix, we’ll update there.
Oct 9, 2015 | Information, Outages, Unplanned downtime
We’ve seen some sessions drop (on ADSL and FTTC/FTTP connections|) following an unexpected port reboot on one of our main switches here.
Most sessions have automatically re-connected but if your line is failing to come back as expected, it may have a stale session which needs to be cleared down. To do this, please power-down your router for 15/20 minutes and then attempt to re-connect.
This should bring you back on-line but please raise a fault with support here in the normal manner if you’re then still having problems.
Oct 8, 2015 | Information, Outages, Unplanned downtime
We are aware of a problem with authentication to our Radius servers in the last 20 minutes that is affecting ONLY circuits delivered as from C20 exchanges — we are working on this now and expect a resolution shortly.
It’s only affecting those lines that have dropped earlier or had been switched off and are now trying to connect (“authenticate”) and not on lines currently logged in all of which should carry on working normally; so is only a problem for a small part of the ADSL estate in Merula.
Aug 7, 2015 | Information, Outages, Unplanned downtime, Update
[Update] 15:45 The engineers have repaired the faults identified and all circuits are up & reporting as being heathy. If anyone is still seeing issues please mail into support@merula.net. We are conducting a postmortem with the supplier to ascertain the root cause, the fix made and to find out why their agreed backup routing didn’t kick in as part of the DR process. We will update the ticket as we know more.
15:10 Multiple back-hauls to us and other customers are affected by this fault; this has been escalated internally by the supplier to their 3rd-level team. No time to fix as yet.
14:26 Engineers have arrived on-site and are starting their investigations.
Engineers are en-route to both ends of this connection.
We are aware of a problem affecting one of our core bearer lines into HEX in London which in turn is affecting a number of customer leased lines. These are currently hard down.
This is a high-priority outage for us and we are working to get this resolved as quickly as possible and apologise for the down-time being seen. We do not currently have a time to fix but expect a status report within the next 30-45 minutes.
We will post updates here.
