Router Upgrade/Reboot 11/04/2020 [completed]

Our core Juniper Router in Harbour Exchange Square has been showing a very low level of alerts and a higher than normal CPU load. This is not service affecting currently.

In order to attempt to clear this we will upgrade the Juniper firmware and reboot it late Saturday evening. This should take no longer than 30 minutes. Most services are dual homes. However a small number of services are directly connected to this router – these will suffer an outage during the reboot.

You may also see ‘different’ routing to some locations during this time as well.

We will update this post as the work progresses

[Update]

This work is now completed with no issues – the firmware upgrade appears to have resolved the issue. We are monitoring overnight

Phishing Email / Text

We have seen a number of email and text messages in a format similar to the below

These messages are not genuine and you should not click on the link – or send any details if you receive these

We have (we believe) blocked this site being reached from the Merula Network – but you may still be able to reach these if you are connected via your phone etc.

We would strongly advise customers to be wary of clicking on any link they are not 100% certain is genuine

 

 

UPDATE: Network outage 3rd Feb 2020 [resolved]

4th Feb: the supplier network resolved all oustabnding issues on this extended outage. Apologies again to anyone affected.

10.35AM UPDATE: this problem which affects a large number of FTTx lines is still being worked on by Openreach engineers. Latest estimate for the work to be completed is 18.30 tonight. We’ll continue to update here as and when we hear anymore news.

———

We are aware that one of our suppliers has a major service outage. The supplier engineers have no ETA to fix as yet. We will; update as we hear more. This will affect ADSL & FTTC lines. Our apologies for this outage.

 

Line affected:

Start: 03/02/2020 04:15 Next update / Cleared: 03/02/2020 09:30 (estimated)
Raised: 03/02/2020 04:15 Cleared Reason: N/A
Detected: 03/02/2020 04:15 Exchange Name: KINGS LANGLEY, LITTLE GADDESDEN, BOLSOVER, STAVELEY, HOLMEWOOD, OLD WHITTINGTON, CLAY CROSS, HOLYMOORSIDE, BASLOW, STONY STRATFORD, HANSLOPE, YARDLEY GOBION, CROXTON, CAXTON, LAISTERDYKE, DUDLEY HILL, UNDERCLIFFE, GRANTON, SOWERBY BRIDGE, ILLINGWORTH, CALDER VALLEY, WOMBWELL,
Incident Headline: Service affecting outage – Loss of Service
Incident Details: Our supplier engineers are working on the fault.
Area Codes: 01131 01132 01133 01138 01162 01173 01179 01212 01213 01214 01215 01216 01217 01223 01226 01246 01257 01274 01275 01311 01312 01313 01314 01315 01316 01317 01332 01422 01442 01446 01454 01480 01509 01773 01827 01908 01923 01926 01942 01954 02010 02032 02033 02081 02082 02083 02084 02085 02086 02087 02089 02476 02911 02920 02921 02922

 

Broadband Issues – 14/1/2020 [Updated]

We have detected a drop in a lot of broadband sessions from one of our upstream wholesale suppliers. We have raised this as a case and will update you as soon as possible. Both of our interconnects are only allowing a small number of live sessions so we believe this issue is on their network.

If you session has sync but will not connect this is probably the issue – we will update this case witin 30 minutes if not before

We apologise for the issue at this time

[update 8:30]

We are now seeing sessions return – we do not have an update from the supplier as yet and are following up for a formal update. As soon as we have some news we will update further

 

Caching Name Server issue [resolved]

We are seeing an issue with one of our caching name servers in London which may be causing slow access to some websites etc. The second name server is up and running and working fine. We are investigating the cause of this issue and will update this post shortly

We are sorry for any issues here

[update] Thos issue should now be resolved. If you are seeing any issues please call or email support as normal

Switch Reboot – London Harbour Exchange Square

One of our core switches in Harbour Exchange Square had a routing issue about 14:20 today – after our team checked the switch it was decided the fastest resolution was to perform a routing engine switchover. We would normally plan this out of hours – However as this was causing issues to customers the decision was taken to perform this immediately.

This affected all services directly connected or routed through this switch

The switch reboot has now completed and service should be restored to all locations and services.

If there is still an ongoing issue for any services please report them to support in the normal way

We apologise for the issues this may have caused

 

OUTAGE: scheduled maintenance DSL overnight 10th April 2019

As part of our commitment to service improvement, we would like to inform you that we will be performing planned critical maintenance as part of capacity upgrades on our core network

Maintenance window start: Wednesday 10th April 2019 23:00 GMT

Maintenance window end: Thursday 11th April 2019 06:00 GMT

Expected impact duration: The work will impact some of our DSL subscriber customers, and they will see an intermittent service within the maintenance, with a short period of down time. Service should be restored automatically but it may require a router re-boot. If after this action, service hasn’t been restored, please raise a support case in the normal manner.

OUTAGE: scheduled maintenance broadband 16th/17th Feb 2019

We will be upgrading some of our core routers that terminate broadband sessions during the course of the weekend – each will be down for less than 30 seconds for an important firmware upgrade. These will be staggered over the two days. Most affected customers will reconnect within seconds automatically. If you have any issues, please reboot your router and if that fails for any reason, please call into or email support in the normal way. This will only affect ADSL, FTTC & FTTP lines.  

OUTAGE: scheduled maintenance broadband 22/1/19

We will be upgrading some of our core routers that terminate broadband sessions later this evening – each will be down for less than 30 seconds for a firmware upgrade. These will be staggered over the evening. Most affected customers will reconnect within seconds automatically. if you have any issues please reboot your router and if that fails for any reason, please call into or email support in the normal way.

This only affects ADSL, FTTC & FTTP lines.

15:42

UPDATE: Avro Court – Power at Risk [resolved]

20/12/18 15:00 UPDATE

The new UPS system has now been installed and commissioned. We now have a full N+1 resilient UPS system in our Avro Court Data Centre. Power is no longer at risk.

19/12/18 10.32am UPDATE

The USP manufacturer has been on-site yesterday and today. After discussions with them, we have decided to take this opportunity to upgrade the current UPS to a later model, giving us increased cover and reliability. This is due to arrive on-site at some point today. Installing, commissioning and testing will however likely roll into tomorrow. 

The power supply to the data centre continues to be at risk until this replacement work is completed.

18/12/18 12:10pm UPDATE:

The engineer has sourced the necessary spares and is again en-route to site. We expect that this work will take the rest of the afternoon with the necessary tests being run and, assuming all goes to plan, the UPS will be brought back on-line during the early evening.

The power supply will be at risk until this is completed.

UPDATE:

The UPS is still being worked on. Whilst we expect this to be completed today, power supply to the data centre should still be considered at risk until this is complete. 

We had a planned service today of our UPS, by specialist engineers. This has uncovered a problem that has caused our main UPS to not come online fully. Currently services in the Data Centre are working fine but the power should be considered at risk until the UPS is fully restored to service.

The Specialist engineers are currently arranging the parts required and once here they will be swapped out. We will update this post as we have more details