Need help or advice?
Call us FREE 0800 298 2375

Network Migration – Ongoing

During the next few weeks we will be migrating our core network onto new, faster Juniper router and switches. The new routers and switches are already in place in our data centres and configured. The current work involves moving links in a controlled manner between the old and new network . The majority of this i moving peering and transit connections one by one. This will not cause any outages – just subtle routing changes as the traffic moves away from a transit – the link is moved and the transit (or peering re-appears via the new network.

Once this is completed there will be some brief blips on leased lines and broadband connections as they are moved to the new network – these will be communicated as these are planed – most outages will be less than a couple of minutes in these cases as the cables are moved.

The end result will be a more modern network with far greater capacity for growth

We will update this site as the plans progress

 

 

Planned Work – HEX Data Centre 11/03/2018 – Completed

As part of the upgrades and improvements to our network we are moving our connectivity in this data centre to a new rack with dual power feeds and additional cooling.

We already have a new router and switch in this rack – However subject to resources we will be moving some of our hardware and links between the two racks this coming Sunday. The work will start in the late afternoon and into the evening.

During this time various parts of our network and services connected directly to this network PoP may experience a period of downtime. Traffic may also take alternative routes to some locations and there’s a chance of some congestion for some traffic.

This PoP currently hosts

– some Transit and peering links which will simply take alternative routes

– One of our DNS servers (the other servers will work fine)

– Some TalkTalk based ADSL traffic – in these cases PPP sessions will drop BUT should auto re-connect on the Telehouse link

– Approx 21 leased line circuits that will be contacted directly

We will confirm the date once we have confirmation of resources from the data centre provider to help move some fibre cross conencts

We will keep the downtime to a minimum and we will update this post as much as we can

Any questions please email support in the normal way

 

[update]

This work is completed. All services in HEX are now in our new rack and core network

Emergency work 15/02/2018

Our supplier has identified a potential issue with a fibre bearer which holds some backup links to our Huntingdon Data Centre. They are making a config change to correct this overnight.

As these are backup links it should not have a significant impact but there is some chance of packet loss as the link drops / returns as the network re-converges.

We will update this post as the work progresses

Planned Work – 9/2/2018 10PM

We have noticed a small issue on a switch port connected to some leased lines. The issue is not currently service affecting – and needs a simple one line config change to resolve.

As there is a very small risk to a few seconds of packet loss during the change this will be performed tonight

Only leased lines connected to a single bearer in Telehouse North will be affected and we don’t expect more than a few seconds of packet loss during the change

Any questions or issues please contact support via the normal means

ADVISORY: security issues with some Netgear routers

Good morning.

Whilst we don’t supply Netgear routers, we know that a number of our customers use them, so wanted to ensure you were aware of potential vulnerabilities that need to be addressed via a firmware update.

The list of (so far) affected models is below. You or your IT specialist should ensure that you update as soon as possible to mitigate any risks.

List of NETGEAR Routers At Risk

DGN2200v4
R6100
R6220
R6250
R6300v2
R6400
R6400v2
R6700
R6900
R6900P
R7000
R7000P
R7100LG
R7300DST
R7500
R7500v2
R7800
D7800
R7900
R8000
R8300
R8500
D8500
WNDR3400v3
WNDR4500v2
EX6200v2

The Five Vulnerabilities

TWSL2018-002: Password Recovery and File Access

Trustwave SpiderLabs Advisory

NETGEAR advisory

Some routers allow arbitrary file reading from the device provided that the path to file is known. Total of 17 products are affected.

TWSL2018-003: Finding 1: Post-Authentication Command Injection

Trustwave SpiderLabs Advisory

NETGEAR advisory

This one affects six products and reflects a root level OS command execution via the device_name parameter on the lan.cgi page, although the attack requires authentication.

TWSL2018-003: Finding 2: Authentication Bypass

NETGEAR advisory

This also affects large set of products (17) and is trivial to exploit. Authentication is bypassed if “&genie=1” is found within the query string.

TWSL2018-003: Chained Attack: Command Injection

NETGEAR advisory

This is a three-stage attack leveraging three separate issues: CSRF token recovery vulnerability and the two findings in TWSL2018-003. As a result, any user connected to the router can run OS commands as root on the device without providing any credentials.

TWSL2018-004: Command Injection Vulnerability on D7000, EX6200v2 and Some Routers

Trustwave SpiderLabs Advisory

NETGEAR advisory

Only 6 products are affected, this allows to run OS commands as root during short time window when WPS is activated.

PLANNED WORK: Core switch upgrades.

Following the rapid growth on the Merula network we have been planning a major upgrade to our core switching platform to Juniper devices.  These will allow us to dramatically increase the capacity between sites and add further stability to our network.  Over the coming week, work will take place to introduce them into our network.
This will in due course lead to some limited downtime as links are migrated. We will update here with specific slots for this but wanted to provide some initial details for this work.
Any initial questions, please email support via the normal means.

PLANNED WORK: 1/2/18 UPDATE

Further to the outage yesterday and the corrective work by the carrier we will attempt to bring the link that failed back into service this evening. If all works to plan there may be a brief network disruption while the network reconverges. If the link fails again we will remove it from the network and continue work with our supplier.

The link was brought on line approx 12:30am this morning – However the previous issue re-occurred. This has been raised to the supplier who believe this is resolved this correctly now, We will plan a further maintenance shortly to bring the link back into service. In the mean time there is no reduction to service – but there is a slight reduction in resilience to our Manchester PoP

OUTAGE: Network drop 31/1/18

During planned work deemed low risk by a supplier they managed to inject a loop into one of our links. This caused a significant level of packet loss into our network at approx 22:50 on 31st January.
The link was removed from use and service was resumed albeit with reduced resilience.  The issue has been reported to the carrier who have identified a potential problem and resolved this.

OUTAGES: connectivity & latency issues

RESOLVED: 13:14pm

We are not aware of any ongoing issues now and believe that the cause of this problem has been identified and remedial action taken. Once again, apologies to anyone affected this morning.

UPDATE:

We have removed one of the backhaul lines from our network as this appears to be causing routing issues; we are seeing the majority of the affected lines coming back to their normal latency and response times.

We will continue to update here and apologise again for this affecting you at the start of the working week.

We are aware of an as yet unidentified issue affecting large numbers of our circuits leading to slow-downs, poor quality links and high latency. We are working on this now and will update here as soon as we have more information to share. We apologise that this is affecting you on a Monday morning.

REMINDER: Contact details

In the event that the phone system here (and the primary support number which routes via here) is unavailable for whatever reason as happened today as our VOIP system failed as part of the outage, there’s a second, fallback (geographical) number: 01480 411616, which rings directly at our support centre.  this should be used.

We also suggest that all customers subscribe to our mailing list ensuring that copies of these status messages and updates are delivered to you by email or alternatively if you prefer, to your favourite RSS reader.

If you have any questions please email support@merula.net.

EMERGENCY CONTACTS

The main support number is 0845 330 0666 (geographical 01480 355566)

There’s a second, fallback (geographical) number: 01480 411616. All numbers ring directly at our support centre, manned 24hrs x 365 days a year.

We'd also suggest that all customers subscribe to our mailing list (link above); status messages and updates will be delivered by email.

Subscribe