Need help or advice?
Call us FREE 0800 298 2375

UPDATE: Avro Court – Power at Risk

18/12/18 12:10pm UPDATE:

The engineer has sourced the necessary spares and is again en-route to site. We expect that this work will take the rest of the afternoon with the necessary tests being run and, assuming all goes to plan, the UPS will be brought back on-line during the early evening.

The power supply will be at risk until this is completed.

UPDATE:

The UPS is still being worked on. Whilst we expect this to be completed today, power supply to the data centre should still be considered at risk until this is complete. 

We had a planned service today of our UPS, by specialist engineers. This has uncovered a problem that has caused our main UPS to not come online fully. Currently services in the Data Centre are working fine but the power should be considered at risk until the UPS is fully restored to service.

The Specialist engineers are currently arranging the parts required and once here they will be swapped out. We will update this post as we have more details

Switch issue – Telehouse West [16/11/18] COMPLETED

We are seeing an issue with one of our Juniper switches in Telehouse West which affects IP addresses directly hosted on the switch. A health check on the swotch shows no issues but routing is impacted.

Currently we have worked round the issue and it is not customer impacting – However we would like to resolve this fully.

As such we will have an engineer on site Friday evening to check and resolve this. We have a spare switch to swap out with if this is a hardware issue

We will update this ticket as the work progresses

Most customers will not notice any issue other than a minor routing change. There are 3-4 directly connected customers who have been notified directly

[update]

This work completed at approx midnight on Friday. The switches have been replaced and the new 10G switches are installed. There was a unexpected issue with Broadband connections terminated in Telehouse East which we are investigating to ensure it does not re-occur

Issues- Manchester

Our Network monitoring has detected a potential power issue at our co-location location in Manchester. Our routers and switches at this location are currently unreachable

We are working with our provider to resolve the cause of this issue an update a soon as we have one. Currently there is no service outage – However this is used as a backup / failover location and as such some services should be considered ‘at risk’ until the service is restored

ADSL Planned Work 18/06/18 – 19/06/18

Overnight on the 18th June – 19th June the link to one of our Wholesale providers is being worked on due to a switch firmware upgrade and reload.

We have dual links to the provider and the work will only affect one line at a time.

During the work (between 10pm on 18th and 6am on the 19th you may see your connection drop as they upgrade and reboot their switches

You should see your connection re-connect  after a few seconds on our other link to them.

If your router does not re-connect automatically please turn your router off for 20 minutes to clear any stale session and then re-connect. If this fails – please call or email support in the normal way. We will update this ticket as we receive updates

 

UPS Maintanance – Avro Court – 18/06/18

As part of our contract with our specialist contractors the UPS in Avro court is having a regular service on 18/08/18.

This involves checking the UPS is functioning correctly, batter health etc

This should not cause any interruption to service but as work is being undertaken here there is always a very small risk.

Any work that is deemed to have a potential of interruption will take place after 5pm. Our own engineers will be on site throughout to monitor the work

We will update this as the work completes

 

Avro Court Upgrades

Following from the upgrades in London and Manchester the final upgrade will be to our Huntingdon Data Centre. We have a number of planned changes over the coming week.

1) Individual Rack Upgrades

A number of our hosting racks have older cisco switches in them – we are planning on upgrading them over this weekend / early over the coming week. We expect this will cause a number of short breaks to service to individual servers. We will have the new rack in place in each rack before and then simply move the patch cable for each server to the new rack. Once all servers are moved to the new switch – the old switch will be removed.

2) Power upgrades

In preparation for the new core switch units we will be adding additional power to our core racks. This should not cause any issues – However our specialist engineers will be working near the main power distribution board as such this should be considered at risk. All work will be carried out outside normal working hours

3) Core Router and Switch replacement

Once the prep work has been completed we will be replacing the core routers and switches with much faster and newer models. During the replacement process this will mean there are short periods of downtime as services and moved to the new switches and routers. We will create a separate ticket for this work as soon as we have a definate date – however we wanted to provide some notice – we anticipate this being later in the week or over the weekend – it will be outside standard working hours.

We will update this ticket as these progress

Manchester Upgrade 16/06 – 17/06 – completed

Continuing the recent network upgrades on our network the core routers and switches in Manchester are due to be upgraded over the weekend to our new Juniper core.

During the upgrade process there will be breaks in access to some of our backup transit connections in Manchester and a number of leased lines connected in Manchester. All leased line connections in Manchester are backup links so we do not expect and connections to customers to go down – However there will be a lack if resilience while this upgrade is in place. We will upgrade this notification as the work progresses

[Update] This work is completed and all services at Manchester have been updated. There was an unplanned issue routing to a small number of locations during this work but this was identified and resolved. If you are seeing any issues please contact support via the normal routes

CLOSED: Portal diagnostics

UPDATE: the supplier code issues have been worked around and all tests are now available again.

—————————–

We are aware of issues working with the APIs from the suppliers that’s affecting the ability to run diagnostics via the Merula portal (https://adslreports.merula.net); we are working on this now and will advise here as soon as this is resolved. Apologies for this loss of service.

OUTAGE: overnight works

RESOLVED: the fibre break was fixed at 8:53pm last night and all services are back to normal. This ticket is now closed.

UPDATE 15:50 — our supplier advises “in relation to the issue identified in Cambridge area regarding loss of service. We are still working hard to resolve these issues as a priority.

Due to heavy traffic in the area, this is impacting our ability to get into the pit location necessary to move services onto spare fibres. Traffic management is required to safely carry out the work and this cannot be implemented until 20:00 this evening due to local authority restrictions.

From 20:00 onwards we can commence repairs so we should see services begin to come back online overnight this evening.

Please accept our apologies at this time as we are treating this matter with the utmost urgency.”

UPDATE: 14:45pm — because of the amount of splicing work needed and the traffic management problems at this location, the ERT has been pushed out to 20.45 today.

UPDATE: 13:00pm — Traffic management is only permitted until 1530Hrs but traffic lights and barriers are on site ready to go. We’ve not been advised of a new ERT as yet.

UPDATE 12:30pm — work continues on-site at the node. Raised for traffic management. Has ERT for 15 minutes time but engineer still working and waiting for response from traffic management team.

Next update due at 1300Hrs.

UPDATE 10:31am — at 1008Hrs an engineer arrived on site. The overnight change was moving to new cable but there appears to be a fibre break. No ERT at present.

UPDATE 10:11am — this is now being treated as a high priority fault as completion has overrun and classified as a MSO due to the affect on Merula (and other) customers through this node.

UPDATE: we are escalating this for a substantive update as the works have again over-run their new completion time.

Emergency unscheduled works by one of our main backhaul suppliers have overrun and mean that one of our 1Gb links to London from the data centre here in Huntingdon is hard down. Traffic is being routed via one of our backhaul links but as this is slower you may see some slowness on some traffic until this work is completed.

We have been told that the work was due to finish at 6am but this has now been pushed out to 8am.  We will continue to update here. Our apologies if this affects your connection(s).

Network Migration – Telehouse North 30/03/2018

Further to the recent posts – we are planning the final Network upgrades in our core network later today. This will decommission the last of our old network in London. This involves the move of our final leased line bearers and the final Broadband Interconnects.

For Broadband Interconnects you may see your session drop – However we have alternative Interconnects to all wholesale providers in our other PoPs so these should re-connect almost instantly on an alternative link.

Leased Lines terminating in Telehouse will have an outage of at maximum 30 minutes while they are moved between the old and new network. In many cases the outages will be much shorter than this

We will update this post as the work progresses

 

 

EMERGENCY CONTACTS

The main support number is 0845 330 0666 (geographical 01480 355566)

There’s a second, fallback (geographical) number: 01480 411616. All numbers ring directly at our support centre, manned 24hrs x 365 days a year.

We'd also suggest that all customers subscribe to our mailing list (link above); status messages and updates will be delivered by email.

Subscribe