Need help or advice?
Call us FREE 0800 298 2375

COVID-19 FIGHT – A MINUTES SILENCE AT 11AM

Just to let you know that Merula — along with many others in the industry — will be observing a minutes silence at 11am today to pay tribute to all the frontline workers who have unfortunately lost their lives in the COVID-19 pandemic.

We appreciate your support.

Network Instability [resolved]

We are aware of some instability on our core London network – this has caused a number of short periods of packet loss this afternoon. Our NOC team are investigating this currently. We will update this as we know more.

This is being worked on as a high  priority currently

We apologise for any issues this may cause

[update 19:10]

Investigations are ongoing on the cause of this issue. Our engineers continue to work to understand and resolve the root cause of this issue. We can see BGP sessions drop between our core routers but do not fully understand the actual cause of this. Many users may not notice the issue however there will be occasional packet loss and ‘strange’ routes while this is being worked on

[Update 20:30]

While this issue has improved this is still ongoing. We are seeing BGP between two of our core routers (Telehouse North and East) flapping. This is causing some packet loss / routing drops. now we have isolated the reason for the drops  we are working to find why this is ongoing. Initial thoughts where that the routers had stale data and we therefore rebooted them – However the issue while improved has not gone away. We will update this further shortly

[Update 23:00]

The network has now been stable for over an hour and we believe the issue is cleared. we will however monitor this overnight before we close this. It appears that this was down to a forwarding issue on a core Juniper Switch in Telehouse West. After ruling out other (more likely) causes, this switch was reloaded approx 22:00 and so far the routing issues and flaps have calmed.

As the switches showed no obvious errors or log issues we will not close this until we are certain – and will investigate in more detail. However we hope there is no further customer issues as a result of this. We are sorry if this issue affected your service today

[Update 26/4/202 10:00]

The network has stayed stable overnight following the reboot of the core Juniper Switch. We are continuing to monitor – however we do believe this issue is now resolved.

We are sorry for any issues caused by this problem

Webmail URL Change – merula.co.uk email addresses

As part of the certificate renewal on our webmail server – the URL for webmail has changed from https://mx1.merula.co.uk to https://mymail.merula.net

The physical server email accounts etc remain the same – it is just the URL for webmail that has changed. If you prefer to access your email from an email client – you can also change the SMTP/IMAP/POP3 servers to the same name – mymail.merula.net

If you do not have a merula.co.uk email account you are not affected by this change

As ever if you have any questions please contact support via the normal routes

 

Rack Issue – Huntingdon 16/4/2020 [update]

There appears to have lost power and/or  a switch failure in a single rack in our Huntingdon Data Centre. This dropped approx 2am this morning. This rack houses a small number of Merula and customer servers

We are aware and will investigate and resolve this asap. We are planning to be on site approx 7am and will resolve this issue then

We apologise for any issues this may cause and will update this as soon as we have more details

 

[update 8:15am]

The issue appears to be related to the switch in the rack – after being offline for approx 90 minutes the switch came back up and connectivity was restored to most servers in the rack. However we are seeing issues with connectivity via a couple of servers in this rack. We have therefore taken the decision to manually reboot the switch to see if this restores service given that the servers themselves look OK and have not rebooted. This will unfortunately result in a loss of connectivity to all services in this rack for a couple of minutes. We will update this as we know more

[Update 8:53]

The switch was rebooted – and the latest saved config has been re-applied. This we believe has restored service to the services we are aware of that had an issue. We are continuing to check for anything else with an issue and are investigating the cause of the switch outage further. We may have to schedule a swap out of the switch if we cannot locate an obvious issue here. However we believe that currently all services in Huntingdon should now be restored. Please do email support if you continue to see any issues

 

[Update 9:20]

This affected switch appears to have failed again. We will now start swapping this out for a replacement switch. We will have an update within the next 45 minutes

[Update 11:30AM]

The switch was replaced and we believe all services have recovered. We are checking for any remaining issues. If you are are seeing any issues please do raise them with support@merula.net. We will update this further later in the day or if we locate any remaining issues

Router Upgrade/Reboot 11/04/2020 [completed]

Our core Juniper Router in Harbour Exchange Square has been showing a very low level of alerts and a higher than normal CPU load. This is not service affecting currently.

In order to attempt to clear this we will upgrade the Juniper firmware and reboot it late Saturday evening. This should take no longer than 30 minutes. Most services are dual homes. However a small number of services are directly connected to this router – these will suffer an outage during the reboot.

You may also see ‘different’ routing to some locations during this time as well.

We will update this post as the work progresses

[Update]

This work is now completed with no issues – the firmware upgrade appears to have resolved the issue. We are monitoring overnight

Phishing Email / Text

We have seen a number of email and text messages in a format similar to the below

These messages are not genuine and you should not click on the link – or send any details if you receive these

We have (we believe) blocked this site being reached from the Merula Network – but you may still be able to reach these if you are connected via your phone etc.

We would strongly advise customers to be wary of clicking on any link they are not 100% certain is genuine

 

 

UPDATE: Network outage 3rd Feb 2020 [resolved]

4th Feb: the supplier network resolved all oustabnding issues on this extended outage. Apologies again to anyone affected.

10.35AM UPDATE: this problem which affects a large number of FTTx lines is still being worked on by Openreach engineers. Latest estimate for the work to be completed is 18.30 tonight. We’ll continue to update here as and when we hear anymore news.

———

We are aware that one of our suppliers has a major service outage. The supplier engineers have no ETA to fix as yet. We will; update as we hear more. This will affect ADSL & FTTC lines. Our apologies for this outage.

 

Line affected:

Start: 03/02/2020 04:15 Next update / Cleared: 03/02/2020 09:30 (estimated)
Raised: 03/02/2020 04:15 Cleared Reason: N/A
Detected: 03/02/2020 04:15 Exchange Name: KINGS LANGLEY, LITTLE GADDESDEN, BOLSOVER, STAVELEY, HOLMEWOOD, OLD WHITTINGTON, CLAY CROSS, HOLYMOORSIDE, BASLOW, STONY STRATFORD, HANSLOPE, YARDLEY GOBION, CROXTON, CAXTON, LAISTERDYKE, DUDLEY HILL, UNDERCLIFFE, GRANTON, SOWERBY BRIDGE, ILLINGWORTH, CALDER VALLEY, WOMBWELL,
Incident Headline: Service affecting outage – Loss of Service
Incident Details: Our supplier engineers are working on the fault.
Area Codes: 01131 01132 01133 01138 01162 01173 01179 01212 01213 01214 01215 01216 01217 01223 01226 01246 01257 01274 01275 01311 01312 01313 01314 01315 01316 01317 01332 01422 01442 01446 01454 01480 01509 01773 01827 01908 01923 01926 01942 01954 02010 02032 02033 02081 02082 02083 02084 02085 02086 02087 02089 02476 02911 02920 02921 02922

 

Broadband Issues – 14/1/2020 [Updated]

We have detected a drop in a lot of broadband sessions from one of our upstream wholesale suppliers. We have raised this as a case and will update you as soon as possible. Both of our interconnects are only allowing a small number of live sessions so we believe this issue is on their network.

If you session has sync but will not connect this is probably the issue – we will update this case witin 30 minutes if not before

We apologise for the issue at this time

[update 8:30]

We are now seeing sessions return – we do not have an update from the supplier as yet and are following up for a formal update. As soon as we have some news we will update further

 

Caching Name Server issue [resolved]

We are seeing an issue with one of our caching name servers in London which may be causing slow access to some websites etc. The second name server is up and running and working fine. We are investigating the cause of this issue and will update this post shortly

We are sorry for any issues here

[update] Thos issue should now be resolved. If you are seeing any issues please call or email support as normal

Switch Reboot – London Harbour Exchange Square

One of our core switches in Harbour Exchange Square had a routing issue about 14:20 today – after our team checked the switch it was decided the fastest resolution was to perform a routing engine switchover. We would normally plan this out of hours – However as this was causing issues to customers the decision was taken to perform this immediately.

This affected all services directly connected or routed through this switch

The switch reboot has now completed and service should be restored to all locations and services.

If there is still an ongoing issue for any services please report them to support in the normal way

We apologise for the issues this may have caused

 

EMERGENCY CONTACTS

The main support number is 0845 330 0666 (geographical 01480 355566)

There’s a second, fallback (geographical) number: 01480 411616. All numbers ring directly at our support centre, manned 24hrs x 365 days a year.

We'd also suggest that all customers subscribe to our mailing list (link above); status messages and updates will be delivered by email.

Subscribe