INFORMATION: Telecity at HEX — work on facility UPS
This is a notification of a possible low “at risk” issue but we believe this is minimal as we also have our own UPS units covering the services & systems located at HEX.
This is the most recent update from HEX:
“UPS works planned have been unsuccessful. Temporary UPS units are available should we need it. Further replacement parts are being sourced and a more detailed action plan is being drafted in order to resolve the issue as soon as possible.
Resiliency level remains at N and there is no anticipated interruption to your services.
We will provide a further update once resolved or if there is any change to the current situation”
COMPLETE: Supplier interconnect work planned 9th Feb 2016
No issues.
Planned Duration:
Start: 09/02/2016 00:01 GMT
Finish: 09/02/2016 05:00 GMT
Task:
One of our interconnect suppliers will be upgrading their capacity with BTW for DSL services in Telehouse North. This will involve migrating traffic on a pair of interconnects to new, higher capacity ones.
Customer Impact:
We do not anticipate any impact to Merula customer traffic, however all BT DSL services delivered from our Telehouse North node should be considered as AT RISK during this maintenance window.
If you have any queries about this work please raise a support ticket with our helpdesk.
We will issue an ALL CLEAR once this work has been completed.
[CLEARED] Speed & traffic issues across the Merula broadband network
Yesterday was Patch Tuesday from Microsoft. In addition, last night Apple released large updates to both iOS and OS X. As a consequence, all supplier back-haul networks (BT, TalkTalk, Vodafone etc) are seeing a dramatic spike in traffic across their links which is impacting our own services to our customers – those of you on ADSL/FTTC type circuits.
We apologise for this but this traffic should slowly die away over the course of the day so download speeds will start to see improvements.
Please bear with us. If you’re still seeing speed problems tonight/tomorrow, then raise a support ticket in the usual manner.
At risk: Electrical supply Avro Court 26th Nov [Completed]
COMPLETED: All Work was completed during the evening on 26th November and the Data Centre now has full access to both Mains Power, Generator Power and the UPS. There was no outage to hosted servers during the work.
UPDATE: we will have engineers on site on Thursday 26th, to replace parts in our automatic transfer panel. Most of the work will not be service affecting, however there will be a short period where the data centre will be supplied by UPS alone. While we don’t anticipate any disruption, the power to our racks should be considered at risk during this period. All “at risk” work will be carried out outside core business hours and we will have staff and external electricians on site as needed to monitor/resolve any issues found.
UPDATE: the engineers are about to isolate the mains supply and generator and move us over to UPS power. This has been tested to support the data centre for at least 30 minutes but for the period of down-time, we are at risk.
We are aware of an issue with our changeover panel here and an engineer is en-route to work on this.
If the panel needs to go off-line then resilience will be reduced for a short period but the generator and UPS are still available and have just been fully tested as part of our normal weekly maintenance regime and will cut in automatically. As a reminder, the generator has fuel for at least 24-hours of continuous running.
Once the panel issue has been resolved, we’ll update this post.
[CLEAR] Telehouse North router maintenance 13th Nov, from 10pm onwards
[UDATE] All work completed to plan.
We plan on making some further changes to a couple of our core network routers to isolate and correct a service setting that is causing a few BGP issues.
This may mean a few blips in routing tables as the changes are disseminated and an occasional up-tick in latency as routes converge and stabilise but we expect no downtime or any significant service issues.
If this planning changes, we will update this ticket