Apr 4, 2022 | Planned Work
We are planning to replace our Juniper EX Series core switches in Telehouse with a newer more powerful model to allow for network growth and increased stability.
We are currently planning on working on the first two sites in Telehouse on Thursday 7th April – we will be working on one site at a time. During the work services will be at risk and access to directly connected connected servers and routers will be affected. Switches will be pre-configured and we hope that the outage per site will not exceed 2 hours. Once we are happy that all woek on one site is complete and the network is stable we will then work on the second site.
In many cases services connected will move to an alternative site – However with some connections including OFNL lines, these will be down for thr duration of the work.
We will update post before and during the work
Mar 28, 2022 | Information, Planned Work
We have seen a few brief drop outs during today for a small number of customers due to a switch in Telehouse West having forwarding issues on a number of core VLANs passing thought that Data Centre.
We’d been planning to announce a switch reboot late this evening to resolve this – however the drops have become more regular this afternoon and we have taken the decision to reboot the switch this afternoon to clear the issue.
We are aware of the cause of the issue (a bug in Juniper OS that sometimes strikes after a config update and had plans to upgrade the switches to more powerful model last week – However as one of our engineers has COVID this work has been slightly delayed.
We can see the switch is now back up and passing traffic – sorry for the lack of notice here but we wanted to restore the stability as quickly as we could and ensure that all services work as they should
Mar 17, 2022 | Planned Work
Over the next week or so customers may see some additional gateways when connecting with their broadband connections.
We have had these installed and tested in our racks for a few months and are following testing we are happy these are ready for customer use. We don’t suspect any issues for customers – however if anything is seen we will just move the connections back to the original gateways.
The New gateways give us extra capacity and should add additional resilience too.
We will let you know how these go – any customers who have questions or think they may have an issue as a result please contact support via the normal routes
Mar 13, 2022 | Planned Work
We have been informed that Virgin Media will be performing a re-route on one of the backhaul links between London and our Huntingdon Data Centre between the 24/3/2022 and 25/3/2022
We have resilient links to provide service here however this will reduce the resilience for services in Huntingdon and also reduces resilience to Manchester
The planned work has a 10 hour window however we don’t anticipate the link being down that long.
We will update this post if we see any impact and once the line is restored.
Mar 13, 2022 | Planned Work
During Monday 14th March our specialist contractors will be replacing the batteries on our UPS in our Huntingdon Data Centre.
While this should have no impact on services – it does pose a slight risk and reduces resilience to services in our Huntingdon Data Centre
We will update as this work progresses
[completed 14:00]
The battery replacement is completed and the UPS and resilience are now back to normal in our Huntingdon Data Centre
Mar 10, 2022 | Unplanned downtime
Unrelated to the issue yesterday – one of our Juniper switches in Telehouse North is reporting errors and is causing routing to toggle between two routes which is causing some latency issues etc.
Under recommendation from our vendor we will reboot the routing engine between 10pm and midnight this evening.
This will affect customers directly connected and will mean that some Broadband customers who terminate in Telehouse may see their connection drop and re-connect to an alternative LNS
We will update this post once the work completes. We are sorry for the short notice but would rather resolve this now before there is any risk of an unplanned outage as a result
Mar 9, 2022 | Information
We are currently seeing an issue with some traffic routing via Telehouse West – we currently believe one of our core switches has a forwarding issue,
Our NOC team are currently working to resolve this which MAY invoice a brief reboot of the affected switch – causing a short downtime for services directly connected to that switch
We apologise for the brief and short notice reboot but want to resolve the issue as quickly as we can
[Update]
The switch is now back from the reboot and we are seeing ttraffic and routing back as it should be – we will investigate further the cause here as a background task to ensure it does not re-occur
Feb 14, 2022 | Planned Work
During the 15th February 2022 engineers will be replacing the fuel on our generator at our Huntingdon Data Centre.
This will not cause any direct issues and we still have our UPS system to maintain power to the Data Centre. It does however mean that in the event of a power failure on the Utility supply there is a risk as for a short period the Generator will not be available to supply load.
Our suppliers will be working hard to ensue that the at risk period is kept to an absolute minimum
NOTE the new fuel will be the more environmentally friendly HVO fuel as part of our commitment to reduce the environmental impact of our company
We will update this post as the work progresses
[Update]
This work completed this morning and the generator is back available as needed
Feb 11, 2022 | Unplanned downtime
We are aware of a Single VM Host currently down which is being worked on now. Our Engineers are migrating the Virtual Machines to an alternative host. We will update this post within 30 minutes with an update
Just an update – this should not affect most users – unless we host a site for you on this particular server. We are currently migrating the data and drives to new hardware and expect to have this back up and running very shortly
The Affected host is now back up and running – the Virtual Machines have been moved to alternative hardware and we see all servers now running as they should. We are sorry for any issues this may have caused.
Feb 1, 2022 | Planned Work
Following on from the fibre break on our Huntingdon to Manchester link yesterday our supplier is carrying out further work overnight to replace approx1km of fibre following the damage they found. This work is currently scheduled to run between 11pm this evening until approx 7am on Wednesday.
We are told the link should not be down for the entire duration but will be down for some point of this.
As previously this will not cause a outage on our network but it does reduce the resilience – and therefore create risk for connectivity at our Huntingdon Data Centre and Manchester.
We will update this post as the work starts and as it progresses / completes
