At Risk – Telehouse West Power Upgrades and Fibre Check 19/5/2023 [completed]

Our engineers along with Telehouse will be working to upgrade the Power Supply to our rack in Telehouse West this evening to allow expansion and additional equipment to be installed. The rack had two resiliant feeds and all core equipment is connected to both feeds.

During the work one Power bar will be replaced at a time – reducing meaning there will for a short period be no resilience on the power in the rack. We do not anticipate any impact but the service should be considered at risk

We also have a fibre link between Telehouse West and Telehouse North which is showing some low level errors. Last week Telehouse tested the fibre which passed – while on site we will replace the optics on this link to see if this clears the issue – if not we will ask Telehouse to run further checks on this fibre. While this is being done again there will be a brief loss of resilience on our fibre ring but we do not expect any customer impact

We will update this if anything changes – and on completion of the work

[completed]

This work completed on Friday evening – we did see two issues through this work. The router that is usd for some OFNL connections for Air customers had an issue for about an hour due to an unplanned powercycle and not re-starting correctly. A few fibre NNIs also bounced. This also affected one of our Name servers. We are sorry for any service disruption from this.

 

 

OFNL Emergency Maintenance 16/03/23 [completed]

With the recent migrations from Air Broadband to Merula we need to perform a router reload and to add additional IP addresses to our routers pool to enable a large number of migrations. This should also clear the issue that has been seen the last few days.

Note this maintenance will only affect customers where we have ‘adopted’ the Air router as opposed to moving to our own setup. We anticipate this will be the last change we need to make for some time.

The plan for this evening is

1) Upgrade Router to latest firmware and reboot
2) Add additional IP Address Pools

Due to the nature of this customers on this router may see a longer initial outage from the upgrade / reboot (approx 10-20 minuites), followed by a few shorter drops as the extra IPs are applied and the router interface reset.

We will start this work shortly and update once complete

 

[completed]
This work is now completed and we can see the sessions that dropped back and we see traffic passing. If you do see any issues – please let us know

 

Leased Line Router Upgrades 20/2/23 and 21/2/23

Our Router vendor has released a new firmware for the routers we install at our customer premises

Over the two dated above we will perform a rolling upgrade to all of our Leased Line customer routers.

Each affected customer will see a 5-10 minute drop on their service while their router upgrades and reboots. Upgrades will start after 10pm each evening and complete in the early hours of the following day.

If you have any questions please do email the support team

At Risk – Network Change 18/2/22 22:00 – 02:00 [completed]

We will make a small network change to update the MTU between a number of our core devices to improve network performance.

This will likely cause a brief blip as the change is applied.

We will update this post as the work completes

[Update 18/2/23]
This work completed just after 10pm yesterday evening and was a sucess

 

At Risk – Network change 20/2/23 22:00 – 02:00 [completed]

We are planning a window to extend our network to two new Data Centres (Cloud House and Slough

There should be no impact on the services, however this is classed as an “At Risk” maintenance window, due to the nature of the works on core network infrastructure.

We will update this post as the work progresses and completes

[Update 21/2/23]
This wok is completed with no impact

 

UPS Swap Avro Court [completed]

We have had an alert that the UPS in one of our racks is failing – this is  discrete UPS as a backup in one of our core cabinets and not the main UPS. We will investigate this and swap out as needed.

Most equipment in the rack is dual fed so there will be no impact. However a few devices are single fed so there may be a small number of devices that briefly power down if the UPS needs swapping out

We plan to do this today as while these is short notice we feel it better to work at the weekend rather than a change or possible issue on a week day.

We will update this post as we are on site and investigate

[Update 13/2/23]
This work completed yesterday the faulty UPS was swapped out and replaced. There was a short downtime to a number of services in Huntingdon as one switch that was used to interconnect several racks was fed from the UPS that needed to be swapped