Jun 22, 2011 | Outages, Planned Work
The below work is now complete. There was a short outage in both SOV and HEX we believe all servers are now working as they are. If you have any issues please contact support.
We are planning on some preventative maintenance work to be undertaken on various items of equipment in our suites at Harbour Exchange Square (HEX) and Sovereign House (SOV) in London.
The works planned are as follows.
At SOV:
(1) Install new UPS batteries as the existing ones are reaching the end of their planned life. This operation is designed to be performed whilst the UPS is running. It should not therefore affect the electrical supply to this cabinet. However, we have a second, spare, tested UPS on-site in case anything does break on the first one and we then lose power.
Services or items that could be affected, if this UPS work fails and we have to fall-back to the spare, lasting for a period of no more than 30 minutes are:
– RADIUS authentication – we have a backup server running in another location, so ADSL users who are not connected before and actually try and connect at this time may find that authentication takes a few seconds longer. No existing connections should be lost or dropped.
– One backup name server will drop off the Merula network; there are 3 others in separate locations. There should be no effect on any clients or other services.
– One server running shared web-sites. All clients on this server who might therefore be affected have already been advised of the work planned.
– One co-lo server. This client has also been advised.
– Merula web-sites: the NOC server, portal server and mymerula sites would all be off-line for this period. The new off-site NOC box mentioned before isn’t due now to go live until early next week as the other ISP is undertaking planned works and can’t go live until then.
(2) Swap out an internal firewall. This will be done at the same time as the above work. The only sites affected will be internal Merula servers. No customer boxes or services are behind this firewall.
(3) Un-rack a couple of obsolete servers. None of these are currently live. No effect on anybody inside or outside the Merula network.
At HEX:
(1) Install a new backup router. No effect on any services or customers. It will be enabled at some future point which we’ll advise in advance.
(2) Replace cabinet UPS batteries and install a new monitoring NIC. Again, this is a hot-swap operation. The UPS should not go down. If problems do arise, again we have a spare, tested UPS on-site and maximum down-time for the services below will be 30 minutes.
– Internal monitoring server and fax server. No external effects.
– Two co-located client servers. These clients have already been advised of this possible down-time.
– One core switch – if power is lost, there will be a small amount of packet loss for 5-10 seconds as the ring stabilises. There is no direct affect on ADSL and connectivity services although there may be a few brief periods where access and traffic may be intermittent as switches & routers pickup new routing due to the work.
If you have not received an email from us, then you should expect to see nothing more, at the worst, than a little bit of instability on your ADSL connection for a period of no more than 30 minutes.
Please do email support@merula.net if you have any concerns or issues about the scope of this work.
Apr 24, 2011 | Outages
Due to a hardware fault on a single server which was temporarily hosting a number of internal systems, some users may have seen an issue on Saturday night.
Systems affected were RADIUS authentication for ADSL (users who were connected were fine, though those that had to reconnect would have had problems), one of our DNS Servers and the Merula webite and portal, since these were all being hosted as virtual servers on the same physical box until the new hardware for each of these, which we were in the process of installing that day, went live.
The cause turned out to be two faulty Hard Drives and appears to be due to a bad batch of drives. Service was restored to some degree by about 2pm and after that we moved the services and servers, one at a time to their new hardware. We believe that all services are now running fully but if you are seeing any remaining issues please call or email support in the normal way.
Apr 7, 2011 | Outages
Last night the mains power failed to the building and kept tripping in and out. The UPS immediately took over and the generator kicked in as planned. Because of the up and down nature of the electrical supply we left the building running on generator until we were happy that the supply company had resolved the power problem which took a number of hours.
Unfortunately, over the last week or so we’ve had approx. 6 similar (albeit lesser) power feed issues and that’s apparently drained the UPS batteries. The engineer attended this morning and advised us that the expected life would be 8-10 years. However, we are now advised after diagnostic tests, that most of the 60-odd cells need replacing now (after only 2-3 years usage). This we’re arranging over the course of the next few days.
The effect of these problems last night was that connectivity with a few servers was lost and not restored until this morning but these we found to be internal issues with dirty file systems not allowing booting and other services on client servers failing to re-start correctly. As far as we can see, apart from the building UPS batteries reaching “end of life” far earlier than we expected, all our systems reacted correctly.
Mar 18, 2011 | Outages
We are aware of some network issues overnight. These afeected connectivity to our Huntingdon Data Centre and some servers in London.
From the information we have it seems that most ADSL sessions that were active were OK but some new sessions did not connect properly until 2am. Initial investigation seems to show a DOS against our network and while we have put in place some blocks for this we are investigating further to get to the root cause and ensure it does not re-occur
Nov 21, 2010 | Outages, Planned Work
BT are working at the exchange that our ADSL central pipe connects to overnight tonight. This is a major upgrade. While we do not have too many details at this point there is a posibility that there may be an outage on our ADSL network between midnight and 6am tomorrow.
This will not affect any of our LLU offerings but may affect any or all of our BT ADSL offerings
Oct 23, 2010 | Outages
Some users may have seen an outage with their ADSL between 5:30 and approx 7:45am this morning. We are still investigating the cause of this issue although the problem appears to be upstream from us at this point. Full service was restored by 8am and we are monitoring the situation to ensure all is well.
If you are still seeing an issue please contact support in the normal way.
