Merula Network Status

We are aware that some users may have seen delays to emails over the last week or two. We have been investigating the cause and now have a plan of action to resolve these once and for all.

Earlier today we made a major change to the config on one of our AV /SPAM servers int he cluster here. Assuimg this is successful the same change will be rolled out to all remaining servers tomorrow and Monday and this should see an end to this issue.

We will update this post as soon as we have more details

During some initial tests on our generator today in advane of the maintanance work on Friday engineers have found a suspect control board. This controls and monitors the generator and is not directly involved in the power feed to our racks.

As part of the work on Friday it is planned to replace this board and resolve the data issue currently being seen. This work should not cause any issues at all, but since this is working on live equiment there is always a risk of an unexpected issue.

We currently don’t have a time for this work to be completed but we will update here as soon as possible

As part of the maintenance at our data centre in Huntington we will have engineers performing maintenance on  one of our generators on the morning of July 11th. There should be no outage or problems related to this but it will reduce resilience  while this takes place.

Due to a huge increase in spam and a hardware failure on one of the mail scanning servers here we are aware that some email has been delayed over the last 24 hours. We now have all servers back up and running and an extra server will be live later this evening.

This issue has also made us bring forward an upgrade to the scanning engine - this has been in testing for a while bit is now live - this should also reduce the amount of spam received - and the accuracy will increase as time goes on.

if you are still seeing any issues please call or email and we will look into the issues as soon as possible

As part of our preperation to perform more diagnostics on one of our routers in Telehouse North following the issues we have been seeing with it we have made some changes to the network configuration. These will also reduce the reliance on this router and therefore mean that any further issues will not be customer affecting.

During the config change you *may* have seen a very brief network issue to some locations however this should not have lasted more than 2-3 minutes at a maximum. All work is now complete and if you are still seeing any issues please contact support in the normal manner.

We apologise for the lack of notice here but wanted to ensure that any furter issue did not cause any problems

Due to a BT incident earlier for which we are still waiting clarification we are currently seeing an imbalance on the central pipes used for the  @adsl.merula and @adsl.wizards realms. if your login is one of the above and you are seeing slow speeds you may like to diconnect and re-login in an attempt to change to the less congested central.

You may need to do this more than once since you have a 50/50 chance od re-connecting to the congested cental.

if we can get moew details from BT we will post them here

Further from the recent router issues we have determined that this is caused by a memory issue on the router hardware. We have now sourced replacement RAM and we will have engineers on site on Friday to replace this. The router itself way be down for 10-15 minutes but most if not all traffic should work round this and there should be little if any noticeable issues

There was a short outage (lasting 10-15 mins) to some servers that are behind one of our core routers (lon3-gw1). Upon investigation it appeared that the router had crashed and was rebooted.

This appears to have restored service and we are now monitoring the router closely - however we do not expect any further issues at this time.

Due to a power failure in one of our London data centres this afternoon some users may have seen issues with some of our services for a short while. Most services were unaffected but thouse routing through Telehouse North may have been affected for approx 30 minutes (many services routed around this issue).

The data centre have confirmed that a number of racks in the suite we are in lost power due to a UPS fault. We are awaiting further assurances that steps have been taken to ensure this issue does not re-occur.

We apologise for any problms caused

It appears that we currently have an a considerable inbalance of ADSL users - with more than 80% of our users on one of our centrals.

If you are seeing slow pings or slow connectivity please try rebooting your ADSL router. You may have to do this more than once since you have about a 1/3rd chance of hitting the same ADSL central on re-connect. If things still look bad later we can ask BT to manually balance the pipes BUT this will cause virtually all sessions to reset - something we would like to avoid if possible