Brief issue in Huntingdon

Due to a stuck route causing an issue with a customer IP range we have just rebooted one of the core routers located in our Huntingdon Data Centre.

This may have lead to a 5 minute window where access to some servers in Huntingdon was intermittent.

All work is now completed and all appears fine. If you have any issues please raise them with support via the nomal means.

[Completed] UPS service work

[UPDATE]

The maintenance work was undertaken with no issues.

_________________________________________________________________________

We apologise for the lack of notice.

The main UPS is being serviced today. We expect no down-time or service issues.

COMPLETED: cgi.merula.net – shared hosting server downtime 18th Aug

[UPDATE] The server move was completed successfully. We believe that all sites are functioning correctly. However, if you think your site has a problem, please email support@merula.net in the normal manner.

[PLANNED WORK] We are seeing a sudden large number of hard drive errors indicating a possible near-term failure on this server which hosts a small number of shared websites. To ensure that this doesn’t cause problems later, we have decided to move this server to a new VM after normal business hours today unless the error rate increases and we need to move more quickly. Down-time of no more than an hour should be expected. We apologise for the short-notice.

Planned maintenance work on transit links 24th August

We have been advised by one of our transit providers that there may be a brief period of instability on some routes between HEX and Manchester early in the morning of 24th August. There should be no loss of service.

Date and Time of Maintenance: 24/08/2011 00:15 – 03:00

Ticket Ref: 120581
Description: Edge-IX Peering LAN Migration
Services Affected: Edge-IX Service

Justification: Expansion of the Edge-IX Peering LAN

Impact: Due to the nature of this work, you may experience a very short disruption in your service during the specified window.

[CLOSED] Connectivity issues

UPDATED:

Connectivity and routing problems occurred yesterday and today which limited connectivity (depending on a couple of strange routing problems inside the both the BT and Level3 networks) for a handful of our customers only. All our testing today seems to show this as cleared now. If you’re still aware of problems, please first switch off your router, wait 30 minutes and restart it. If after this you still need assistance, please email support@merula.net

We are in the process of re-routing around the Level3 network.

We’re aware of a few reports of connectivity issues, either failed name resolution or ADSL connection failures. We’re currently unaware of the reason but will update here as soon as we know more. Apologies for this.

ADSL issues [resolved]

UPDATE: This appears to have been an issue inside BT which has now been cleared by them.

We are seeing a few reports of users unable to login or authenticate ADSL sessions. We can see no problems here at the moment but are investigating further. If you are having problems, please email support@merula.net or call into the support line.

Some servers are down [resolved]

A core internal Cisco switch just died. We are waiting for the replacement to “learn” the current network and this should be in place within approx. 20 mins. Currently, this means that following servers are inaccessible and next to them is the effect this may have. Please email support@merula.net if you have any questions.

Radius authentication: Possible slow logins on ADSL
DNS backup server: Should have very little effect
One backup mail server Possible slight delays in mail being delivered.
One shared web server The sites hosted on here will be down

Maintenance work in our London suites on 23rd June [completed]

The below work is now complete. There was a short outage in both SOV and HEX we believe all servers are now working as they are. If you have any issues please contact support.

We are planning on some preventative maintenance work to be undertaken on various items of equipment in our suites at Harbour Exchange Square (HEX) and Sovereign House (SOV) in London.

The works planned are as follows.

At SOV:

(1) Install new UPS batteries as the existing ones are reaching the end of their planned life. This operation is designed to be performed whilst the UPS is running. It should not therefore affect the electrical supply to this cabinet. However, we have a second, spare, tested UPS on-site in case anything does break on the first one and we then lose power.

Services or items that could be affected, if this UPS work fails and we have to fall-back to the spare, lasting for a period of no more than 30 minutes are:

– RADIUS authentication – we have a backup server running in another location, so ADSL users who are not connected before and actually try and connect at this time may find that authentication takes a few seconds longer. No existing connections should be lost or dropped.

– One backup name server will drop off the Merula network; there are 3 others in separate locations. There should be no effect on any clients or other services.

– One server running shared web-sites. All clients on this server who might therefore be affected have already been advised of the work planned.

– One co-lo server. This client has also been advised.

– Merula web-sites: the NOC server, portal server and mymerula sites would all be off-line for this period. The new off-site NOC box mentioned before isn’t due now to go live until early next week as the other ISP is undertaking planned works and can’t go live until then.

(2) Swap out an internal firewall. This will be done at the same time as the above work. The only sites affected will be internal Merula servers. No customer boxes or services are behind this firewall.

(3) Un-rack a couple of obsolete servers. None of these are currently live. No effect on anybody inside or outside the Merula network.

At HEX:

(1) Install a new backup router. No effect on any services or customers. It will be enabled at some future point which we’ll advise in advance.

(2) Replace cabinet UPS batteries and install a new monitoring NIC. Again, this is a hot-swap operation. The UPS should not go down. If problems do arise, again we have a spare, tested UPS on-site and maximum down-time for the services below will be 30 minutes.

– Internal monitoring server and fax server. No external effects.

– Two co-located client servers. These clients have already been advised of this possible down-time.

– One core switch – if power is lost, there will be a small amount of packet loss for 5-10 seconds as the ring stabilises. There is no direct affect on ADSL and connectivity services although there may be a few brief periods where access and traffic may be intermittent as switches & routers pickup new routing due to the work.

If you have not received an email from us, then you should expect to see nothing more, at the worst, than a little bit of instability on your ADSL connection for a period of no more than 30 minutes.

Please do email support@merula.net if you have any concerns or issues about the scope of this work.

UPDATE: Additional support routes

As many of you know, this site is currently the main route for notifications of outages or issues (whether planned or not).

This won’t change but we are now making a number of additional improvements to ensure that updates are available as soon as possible and, importantly, are available at all times.

The first stage in this work is to re-locate the noc server to a dual-homed location in Manchester so that (nuclear attack aside) this will be immune to any issues affecting Huntingdon and Merula and will ensure people can receive updates even if the data centre itself is unavailable for whatever reason.

We know that last night the noc server was down for a while during the outage, which, whilst planned for and announced, is less than ideal. In future, this won’t be the case.

We’ll also be changing slightly the layout, removing obsolete links and making the site easier & quicker to navigate.

There’s a very useful option available on the noc site to sign-up for an RSS feed or (shortly) an email notification which will mirror our postings here and we strongly suggest that all of our customers take advantage of this facility.

We’ll also be adding a feed from BT of their notifications to us of work & outages which will be displayed there, which will allow customers to check if this is a local, major BT or Merula related issue. This is currently available here but soon will be part of a “dashboard” of notifications on the main page.

We’ve setup a Twitter account to be fed by (or feed to) the noc site, so you can follow that account as well http://twitter.com/MerulaSupport.

We’ll also be continuing to update the telephone support number status message which should be used as the primary support route.

As a reminder to everybody, the normal support number is 0845 330 0666.

Finally, we’ve today setup a second, fallback support number which rings directly at our support centre, to be used in the event that the phone system here (and support number which routes from here) is unavailable for whatever reason. Please take a note of this new number: 0844 2011771