Between 23:00 and 05:00 on the night of 14th running into 15th, one of our back-haul providers is making some configuration changes. At some point during this window there may be a brief (30 second or so) increase in lost packets but there should be no loss of Internet connectivity. During this same brief window, the connections to any servers hosted at our Manchester POP will be at risk.
We are seeing packet loss on one of the 3 links from our Huntingdon Data Centre to London. We have shut the link down which has eliminated the packet loss – BUT has left slightly less resilience on our network.
Late this evening we will be re-enabling the link and running some tests – enabling us to report the issue to our provider as needed. During the tests there may be short periods with reduced access (or packet loss) to our servers in Huntingdon
We apologise for any issues caused
Our supplier is undertaking their regular maintenance & checking work on our main UPS at the Huntingdon data-centre on this date. There should be no impact on the day to day operations and this is informational only.
Following the problems we’re still seeing as part of the over-running planned upgrade to our email infrastructure, we’ve decided that it’s necessary to take the mail-servers off-line at 5pm today for a period of up to 2 hours to allow this syncing to proceed quickly to completion.
Once that has occurred and run to completion, we’re confident that the new system will respond correctly as planned.
Update: The primary server is now back in service with all emails – We will be working on the backup server tomorrow – however we do not expect this to be service affecting.
Update: The mail servers are now back as a resilient pair – initial testing has shown this to be much faster and reliable – any issues please contact support
[UPDATE]
Update: The mail servers are now back as a resilient pair – initial testing has shown this to be much faster and reliable – any issues please contact support
22nd April, 10:53am
We’re aware of slow-downs affecting some email users and are making some “behind the scene changes” to prioritise sending & receiving emails during the day, deferring the syncing of the old email archive until the early hours. We believe that this final sync will be finished by tomorrow at which point the full effects of the new hardware will be apparent.
10.30am — We are in touch with the supplier of the mail server software, as the process is taking far longer than they indicated would be needed. We apologise for this degraded service, which based on the progress so far, seems likely to continue being less than ideal for the rest of the day.
09:00AM The synching process is taking longer than we’d hoped, so whilst this is continuing the symptoms mentioned below will continue. Email access is currently slow as these processes carry on.
We are working to upgrade two of our primary mail servers to new, faster hardware over the next few days. During the early hours of the morning (one each day) one of our mail servers will be replaced.
This will lead to a short downtime while the server is physically replaced and a period while the new mail server syncs its data from the old server. During this time you may find that some mail is not displayed if you use IMAP or Web to collect email. No email will be lost and once the sync completes all email will be visible again.
Once we are sure the first server is in place and behaving we will repeat the exercise replacing the second server.
We will update this post as the work progresses
We had to reboot one of our core ADSL routers as we were aware of various problems affecting a significant number of lines and we took the decision that a very short outage was needed to allow these sessions to be cleared down. We have seen all of the affected lines come back successfully, so believe that this is now resloved. Apologies to anyone who briefly lost their connection.
The porting has now completed and we believe everything is now routing correctly. Please notify us as below if you become aware of any problems.
_______________________________________________________________
Tomorrow we port our numbers to a new provider which may mean that, for a short window during the working day, the main 01480 numbers cease to be routed correctly.
Normal service will resume after this window; unfortunately we don’t know from the supplier exactly what time this will process will occur but down-time should be minimal.
If you should need to call in and find that the number is unobtainable or isn’t answered, please call our temporary number:
01480 708010
or email support@merula.net
There may be a brief (max. 15 minute) loss of connectivity to servers & services hosted at Avro Court on this date as one of our back-haul providers (Virgin Media) upgrades their connections in London and locally. This work is scheduled for the early hours of the morning on 22nd and we will be monitoring this at all the way through the maintenance window to ensure that this completes successfully and that there are no residual issues.
We have been told that this work will occur between 1am & before 6am.
From monitoring previous such work we expect our network resilience to mean that there will be no loss of connectivity — we have flagged this as a possible “at risk” however.
[UPDATE] Work completed successfully late yesterday afternoon. We’re now back at N+1 on the air-con units.
One of our two redundant air-con units has developed a fault; the second unit kicked in automatically and the everything is running correctly but until this unit is fixed (we are waiting for parts to be delivered for the engineer to install) we running at less than full redundancy.
UPDATE: the electrical company replaced the sub-station fuse & the phase came back on-line.
There is currently a power outage affecting the area around our Data Centre in Huntingdon. The generator is providing power as it should – however services should be considered at risk.
UK Power networks are aware and have an engineer on route to replace what they believe is a blown fuse at the local substation
We will update this as we know more.