UPGRADE – the.ibr1 – 27/10/2021

We have an issue with the above router that is causing some issues on our monitoring – and with a couple of internal BGP sessions being down.

We are planning to upgrade and resolve this during a maintenance this evening.

Given the redundancy we now have we do not anticipate this work being noticeable – however it does reduce slightly the resilience within our network while the work is performed. We intend to start this work approx 9pm and it should be completed within approx 1 hour – thought we have a window until midnight just in case.

We will update this post once the work completes

Router Upgrade 19/10/2021

Following on from the packet loss seen earlier today our vendor has suggested an upgrade to be actioned to help resolve the issue we saw.

Given the fault we have decided to start the upgrade process later this evening. This will be approx 9pm – 10pm. As this will be in a controlled manner this should have minimal – if any impact to customer services but there may be brief periods where traffic takes alternative routes to some destinations. While we would prefer to provide more notiece however given the issues seen we would rather ensure this is completed so any follow up work can progress quickly.

We will update this ticket later this evening with an update on the progress

 

Earlier packet Loss [19/10/2021]

We are aware of an issue that caused a period of packet loss and potentially slow connecticity for services located or Routed via Telehouse East earlier today.

The network is now stable and not showing any issues. We are working with the Data Centre and the NOC team to fully understand the trigger for this period of instability. IF any work is needed following on from this we will update this ticket.

 

UPDATE: Outage on some broadband lines [13/10/2021]

14:46 UPDATE
The supplier advises their issues should now have been resolved, however it may take some time for all circuits to reconnect due to the increased load of subscribers attempting to login to their Radius. Our apologies again for this supplier outage.
13.20 UPDATE
Good afternoon.
The supplier had a power outage at one of their data centres that’s affected their network transit and backhaul and therefore some

lines. We are seeing some lines return but not all and will update here as we get more news.

————-
Good afternoon.
We are aware of an issue within one of our suppliers we use for these lines. At this point we don’t have an announcement or full details but we are aware of this. The issue is also affecting their portal which we access and their status page isn’t working.
We are trying to get an update on this which we will pass on as soon as we can.
Note: this issue is not on the Merula network but within the supplier network. Our apologies to this circuits affected.

Phone Issues

Due to an on-going incident with one of our SIP providers we are aware that calls to some of our own numbers may have call quality issues, The supplier is working to resolve this. In the mean time if you wish to call us please feel free to call us on 0330 341 6666 for support or  0330 341 0000 for our main number which uses an alternative carrier.

This will still reach the same teams here to help with any query

 

Network Issue [26/09/2021] – RESOLVED

We are aware of a network issue that is causing connectivity issues for some customers currently – the effect depends which PoP your connection is on.

Our engineers are aware and have been working to resolve this urgently. We expect to have an update in the next 30 minutes

[update 2:30pm]
We believe the vast majority of the packet loss and issues seen has now cleared.We are aware of a few sites that can’t be reached and there is ongoing work – with the potential for planned work later today. If you do have a site you can’t access etc please do email support or call in since the extra information will ensure that we check and clear all remaining issues.

[update 9:30pm]
Work on the network has now finished for tonight. The network has remained stable most of the day with the exception on a small number of sites which showed issues. Troubleshooting has been performed this evening and passed to the router vendor to help locate the underlying issue. However we do understand the trigger and have worked round these for now. If you are still seeing any issues please let us know. We will close this issue shortly and raised a planned work as needed in due course,

This issue is now fully resolved and out network is now live on our new network core

Network Upgrades – London 25/9/2021

We will be installing additional core routers in out Harbour Exchange Data Centre on Saturday.

During most of the work there will be no impact to any services – However there may be brief periods during the day when traffic takes slightly different routes than normal. In addition we will be upgrading one of our two LINX peering connections over the weekend (the first was upgraded yesterday). As part of this the link being upgraded will be shut down this evening and will be re-enabled over the weekend. Given the other link is now upgraded there will be no traffic congestion but you may see slightly different routes to some locations.

We will update this as the work progresses

Routing Issue – 24/9/2021

We are aware of a routing issue affecting traffic to some destinations currently. The NOC team are aware of this and are working to resolve this – we expect this to be resolved very shortly and will update as we know more

[Update 8:30]
We believe the reachability issue is now resolved – there is some further work taking place to follow up – however this will not have any impact at this time. We will work to bring the failed link back to restore redundancy shortly however this will not have any impact on services at this time.

Network Upgrades – Telehouse North – 31/7/2021 – COMPLETED

On the 31st July we will be performing some upgrades to our network in Telehouse North

This will involve installing a new 10G Juniper Switch and installing an additional core router. The work is planned and there should be minimal disruption to services, We will be moving traffic away from the impacted links as the work progresses. This will mean that there will be periods while you may see different routes for traffic.

There is the chance of some very short periods of network disruption when some core fibre links are moved – but again this should only impact a small number of customers who have been advised in avance

This work is part of a rolling program that will significantly enhance the capacity and reliability of our network

We will be updating this post as the work progresses

[Update 1/8/2021]

The install of the 10G Switch is completed. We did not manage to install the new router due to some issues seen – we will re-pan this work for the next few days and update here once it is scheduled

Planned Work – Telehouse East Router 23/05/2021 22:00 – 23:59 – COMPLETED

Our core Juniper Router in Telehouse East is beginning to show high Memory Utilisation. We will therefore perform a planned software upgrade and Reboot on this router on Sunday Evening. Expected downtime is approx 20 minutes.

In most cases the actual outage will be much shorter (or not noticed) since many services will route round the affected router.

However directly connected customers may see a full outage for the 20 minutes of the router reboot. We will update as this completes

[Update 23/5/2021 – 23:40]
This work has completed as planned