Mail Server Maintenance / Upgrade [complete]

[UPDATE]

Update: The mail servers are now back as a resilient pair – initial testing has shown this to be much faster and reliable – any issues please contact support

22nd April, 10:53am

We’re aware of slow-downs affecting some email users and are making some “behind the scene changes” to prioritise sending & receiving emails during the day, deferring the syncing of the old email archive until the early hours. We believe that this final sync will be finished by tomorrow at which point the full effects of the new hardware will be apparent.

10.30am — We are in touch with the supplier of the mail server software, as the process is taking far longer than they indicated would be needed. We apologise for this degraded service, which based on the progress so far, seems likely to continue being less than ideal for the rest of the day.

09:00AM The synching process is taking longer than we’d hoped, so whilst this is continuing the symptoms mentioned below will continue. Email access is currently slow as these processes carry on.

We are working to upgrade two of our primary mail servers to new, faster hardware over the next few days. During the early hours of the morning (one each day) one of our mail servers will be replaced.

This will lead to a short downtime while the server is physically replaced and a period while the new mail server syncs its data from the old server. During this time you may find that some mail is not displayed if you use IMAP or Web to collect email. No email will be lost and once the sync completes all email will be visible again.

Once we are sure the first server is in place and behaving we will repeat the exercise replacing the second server.

We will update this post as the work progresses

 

Some ADSL sessions may have just dropped

We had to reboot one of our core ADSL routers as we were aware of various problems affecting a significant number of lines and we took the decision that a very short outage was needed to allow these sessions to be cleared down. We have seen all of the affected lines come back successfully, so believe that this is now resloved. Apologies to anyone who briefly lost their connection.

[UPDATE] Calling us on Tuesday 14th April

The porting has now completed and we believe everything is now routing correctly. Please notify us as below if you become aware of any  problems.

_______________________________________________________________

Tomorrow we port our numbers to a new provider which may mean that, for a short window during the working day, the main 01480 numbers cease to be routed correctly.

Normal service will resume after this window; unfortunately we don’t know from the supplier exactly what time this will process will occur but down-time should be minimal.

If you should need to call in and find that the number is unobtainable or isn’t answered, please call our temporary number:

01480 708010

or email support@merula.net

 

[UPDATE] Possible loss of connectivity to hosted servers at Avro Court 22nd April

There may be a brief (max. 15 minute) loss of connectivity to servers & services hosted at Avro Court on this date as one of our back-haul providers (Virgin Media) upgrades their connections in London and locally. This work is scheduled for the early hours of the morning on 22nd and we will be monitoring this at all the way through the maintenance window to ensure that this completes successfully and that there are no residual issues.

We have been told that this work will occur between 1am & before 6am.

From monitoring previous such work we expect our network resilience to mean that there will be no loss of connectivity — we have flagged this as a possible “at risk” however.

[COMPLETED] Aircon – Avro Court

[UPDATE] Work completed successfully late yesterday afternoon. We’re now back at N+1 on the air-con units.

One of our two redundant air-con units has developed a fault; the second unit kicked in automatically and the everything is running correctly but until this unit is fixed (we are waiting for parts to be delivered for the engineer to install) we running at less than full redundancy.

Cleared at 12:29 At Risk – Avro Court

UPDATE: the electrical company replaced the sub-station fuse & the phase came back on-line.

There is currently a power outage affecting the area around our Data Centre in Huntingdon. The generator is providing power as it should – however services should be considered at risk.

UK Power networks are aware and have an engineer on route to replace what they believe is a blown fuse at the local substation

We will update this as we know more.

 

Network ‘Blip’ 12/03 12:10 am

We have just seen a short outage with routing within our Network – primarily to our Avro Court Data Centre.

This seems to have been caused by one of the core links flapping between Telehouse and Huntingdon. The link appears to have stabilised and we are monitoring for any ongoing issue here.

We are also liaising with one of our fibre providers to see if they are aware of the issue

Core router swap-out Due: 13th March 2015 Time: 10pm onwards

We are upgrading & replacing one of our resilient core Cisco routers in Telehouse late on Friday night.

Whilst we do not expect any down-time on any of our circuits related to this work, you should be aware that there may be temporary routing issues for a period up to an hour, whilst this is being completed.

UPDATE: delayed At Risk – Generator Test / Service 21/01/2015

This test has now been delayed until 21st.

During the day of 12th January 2015 we are having a full service on our generator including a load bank test which will stress test our generator and ‘prove’ it is fully working and able to take the load from our data centre. This will not affect the ‘normal’ usage in our data centre.

However at times during the work this will mean the generator is not able to take the load IF a power issue occurs. We do still have our on site generator and we are not anticipating any issues – however as there is a small risk we are publishing this in advance.

We will update this post once the work is complete