Sep 26, 2021 | Uncategorized
We are aware of a network issue that is causing connectivity issues for some customers currently – the effect depends which PoP your connection is on.
Our engineers are aware and have been working to resolve this urgently. We expect to have an update in the next 30 minutes
[update 2:30pm]
We believe the vast majority of the packet loss and issues seen has now cleared.We are aware of a few sites that can’t be reached and there is ongoing work – with the potential for planned work later today. If you do have a site you can’t access etc please do email support or call in since the extra information will ensure that we check and clear all remaining issues.
[update 9:30pm]
Work on the network has now finished for tonight. The network has remained stable most of the day with the exception on a small number of sites which showed issues. Troubleshooting has been performed this evening and passed to the router vendor to help locate the underlying issue. However we do understand the trigger and have worked round these for now. If you are still seeing any issues please let us know. We will close this issue shortly and raised a planned work as needed in due course,
This issue is now fully resolved and out network is now live on our new network core
Sep 24, 2021 | Planned Work
We will be installing additional core routers in out Harbour Exchange Data Centre on Saturday.
During most of the work there will be no impact to any services – However there may be brief periods during the day when traffic takes slightly different routes than normal. In addition we will be upgrading one of our two LINX peering connections over the weekend (the first was upgraded yesterday). As part of this the link being upgraded will be shut down this evening and will be re-enabled over the weekend. Given the other link is now upgraded there will be no traffic congestion but you may see slightly different routes to some locations.
We will update this as the work progresses
Sep 24, 2021 | Outages
We are aware of a routing issue affecting traffic to some destinations currently. The NOC team are aware of this and are working to resolve this – we expect this to be resolved very shortly and will update as we know more
[Update 8:30]
We believe the reachability issue is now resolved – there is some further work taking place to follow up – however this will not have any impact at this time. We will work to bring the failed link back to restore redundancy shortly however this will not have any impact on services at this time.
Jul 30, 2021 | Uncategorized
On the 31st July we will be performing some upgrades to our network in Telehouse North
This will involve installing a new 10G Juniper Switch and installing an additional core router. The work is planned and there should be minimal disruption to services, We will be moving traffic away from the impacted links as the work progresses. This will mean that there will be periods while you may see different routes for traffic.
There is the chance of some very short periods of network disruption when some core fibre links are moved – but again this should only impact a small number of customers who have been advised in avance
This work is part of a rolling program that will significantly enhance the capacity and reliability of our network
We will be updating this post as the work progresses
[Update 1/8/2021]
The install of the 10G Switch is completed. We did not manage to install the new router due to some issues seen – we will re-pan this work for the next few days and update here once it is scheduled
May 16, 2021 | Uncategorized
Our core Juniper Router in Telehouse East is beginning to show high Memory Utilisation. We will therefore perform a planned software upgrade and Reboot on this router on Sunday Evening. Expected downtime is approx 20 minutes.
In most cases the actual outage will be much shorter (or not noticed) since many services will route round the affected router.
However directly connected customers may see a full outage for the 20 minutes of the router reboot. We will update as this completes
[Update 23/5/2021 – 23:40]
This work has completed as planned
May 13, 2021 | Planned Work
We are seeing a number of MAC address learning issues flagged on the logs of one of our core Juniper switches in the Huntingdon Data Centre
This is currently not service affecting – However we would like to clear this before it causes an issue. As such we will upgrade the firmware on the switch and reboot it later this evening.
This will affect access to some services in the Huntingdon Data Centre and some leased lines which are patched via it.
We expect the outage to be no longer than 20 minutes and we do have spare hardware etc on site should anything not go as planned.
We will update this post as the work progresses
We apologise for any issues that you may have as a result
This work was completed we did have an issue with accessing the switch during the upgrade but service disruption to customers was minimal.
May 9, 2021 | Unplanned downtime
We are aware of an issue with one of our Juniper Core Routers in Harbour Exchange Square.
The vast majority of services have routed round this and are not affected. As small number of directly connected customer may be seeing an issue
Our Engineers are working to restore service to this router. We will update this as we know more
[Update 18:40]
A remote hands reboot of the core router has not restored service. Our Engineers are therefore en-route to the data centre to investigate and restore service – we expect them to be on site by approx 20:15 this evening. We will update as soon as they arrive.
[Update 20:15]
Our Engineer is approx 20 minutes from the Data Centre they have collected spare parts on route in case any hardware needs swapping out. Next update by 21:15
[Update 21:15]
Our engineer is on site and working on the core router. We can see the file system is corrupt which means the router did not boot when power cycled. We are working to restore this asap. In parallel we have moved some lines onto the switch on site which is working. We are moving other links as well to restore their service. Next update within an hour
[Update 22:45]
The Router is now booted after the disk corruption was cleared. Config is being copied back & applied. We hope to have service restored very shortly. Next update by 23:30
[Update 23:20]
The router is now back and passing traffic – we have not yet enabled all peers so some traffic may take a slightly different route to ‘normal’ but no customer services are now impacted. As part of the recovery the software upgrade planned for this router has been applied so the planned work for that upgrade is no longer needed
[Update 23:30]
During the checking process we have detected the need for a reboot to ensure the router and config are updated completely. The reboot is in progress and we will expect this to complete in 10-20 minutes
[Update 23:50]
The reboot cleared the router alarm and all routing is now back up. Monitoring is now showing all links working as they should be. There will be some further checks to complete – however we do not expect any further issues. We apologise for any customers affected by the issues this evening.
May 9, 2021 | Planned Work
Our core Juniper Router in Harbour Exchange Square is beginning to show high Memory Utilisation. We will therefore perform a planned software upgrade and Reboot on this router on Monday Evening. Expected downtime is approx 20 minutes.
In most cases the actual outage will be much shorter (or not noticed) since many services will route round the affected router.
However directly connected customers may see a full outage for the 20 minutes of the router reboot. We will update as this completes
This work has been cancelled – the router was upgraded during the issue on Sunday
May 9, 2021 | Planned Work
Our core Juniper Router in Telehouse East is beginning to show high Memory Utilisation. We will therefore perform a planned software upgrade and Reboot on this router on Sunday Evening. Expected downtime is approx 20 minutes.
In most cases the actual outage will be much shorter (or not noticed) since many services will route round the affected router.
However directly connected customers may see a full outage for the 20 minutes of the router reboot. We will update as this completes
Note due to the issue in Harbour Exchange Square this work will not take place tonight. A new date will be advised shortly
Apr 18, 2021 | Planned Work
Our core Juniper Router in Telehouse North is beginning to show high Memory Utilisation. We will therefore perform a planned software upgrade and Reboot on this router on Monday Evening. Expected downtime is approx 20 minutes.
In most cases the actual outage will be much shorter since many services will route round the affected router.
However directely connected customers may see a full outage for the 20 minutes of the router reboot. We will update as this completes
[Update 20/4 01:47]
The Router is now rebooted and upgraded. We had a slight delay due to waiting for a handover of one ccross connect which did not complete until just before midnight
There are some final routing changes being made now to enable all upstream links however the network should now be stable
