Need help or advice?
Call us FREE 0800 298 2375

Outages

Network Instability [resolved]

We are aware of some instability on our core London network – this has caused a number of short periods of packet loss this afternoon. Our NOC team are investigating this currently. We will update this as we know more.

This is being worked on as a high  priority currently

We apologise for any issues this may cause

[update 19:10]

Investigations are ongoing on the cause of this issue. Our engineers continue to work to understand and resolve the root cause of this issue. We can see BGP sessions drop between our core routers but do not fully understand the actual cause of this. Many users may not notice the issue however there will be occasional packet loss and ‘strange’ routes while this is being worked on

[Update 20:30]

While this issue has improved this is still ongoing. We are seeing BGP between two of our core routers (Telehouse North and East) flapping. This is causing some packet loss / routing drops. now we have isolated the reason for the drops  we are working to find why this is ongoing. Initial thoughts where that the routers had stale data and we therefore rebooted them – However the issue while improved has not gone away. We will update this further shortly

[Update 23:00]

The network has now been stable for over an hour and we believe the issue is cleared. we will however monitor this overnight before we close this. It appears that this was down to a forwarding issue on a core Juniper Switch in Telehouse West. After ruling out other (more likely) causes, this switch was reloaded approx 22:00 and so far the routing issues and flaps have calmed.

As the switches showed no obvious errors or log issues we will not close this until we are certain – and will investigate in more detail. However we hope there is no further customer issues as a result of this. We are sorry if this issue affected your service today

[Update 26/4/202 10:00]

The network has been stable overnight following the reboot of the core Juniper Switch. We are continuing to monitor – however we do believe this issue is now resolved.

We are sorry for any issues caused by this problem

UPDATE: Network outage 3rd Feb 2020 [resolved]

4th Feb: the supplier network resolved all oustabnding issues on this extended outage. Apologies again to anyone affected.

10.35AM UPDATE: this problem which affects a large number of FTTx lines is still being worked on by Openreach engineers. Latest estimate for the work to be completed is 18.30 tonight. We’ll continue to update here as and when we hear anymore news.

———

We are aware that one of our suppliers has a major service outage. The supplier engineers have no ETA to fix as yet. We will; update as we hear more. This will affect ADSL & FTTC lines. Our apologies for this outage.

 

Line affected:

Start: 03/02/2020 04:15 Next update / Cleared: 03/02/2020 09:30 (estimated)
Raised: 03/02/2020 04:15 Cleared Reason: N/A
Detected: 03/02/2020 04:15 Exchange Name: KINGS LANGLEY, LITTLE GADDESDEN, BOLSOVER, STAVELEY, HOLMEWOOD, OLD WHITTINGTON, CLAY CROSS, HOLYMOORSIDE, BASLOW, STONY STRATFORD, HANSLOPE, YARDLEY GOBION, CROXTON, CAXTON, LAISTERDYKE, DUDLEY HILL, UNDERCLIFFE, GRANTON, SOWERBY BRIDGE, ILLINGWORTH, CALDER VALLEY, WOMBWELL,
Incident Headline: Service affecting outage – Loss of Service
Incident Details: Our supplier engineers are working on the fault.
Area Codes: 01131 01132 01133 01138 01162 01173 01179 01212 01213 01214 01215 01216 01217 01223 01226 01246 01257 01274 01275 01311 01312 01313 01314 01315 01316 01317 01332 01422 01442 01446 01454 01480 01509 01773 01827 01908 01923 01926 01942 01954 02010 02032 02033 02081 02082 02083 02084 02085 02086 02087 02089 02476 02911 02920 02921 02922

 

Broadband Issues – 14/1/2020 [Updated]

We have detected a drop in a lot of broadband sessions from one of our upstream wholesale suppliers. We have raised this as a case and will update you as soon as possible. Both of our interconnects are only allowing a small number of live sessions so we believe this issue is on their network.

If you session has sync but will not connect this is probably the issue – we will update this case witin 30 minutes if not before

We apologise for the issue at this time

[update 8:30]

We are now seeing sessions return – we do not have an update from the supplier as yet and are following up for a formal update. As soon as we have some news we will update further

 

Switch Reboot – London Harbour Exchange Square

One of our core switches in Harbour Exchange Square had a routing issue about 14:20 today – after our team checked the switch it was decided the fastest resolution was to perform a routing engine switchover. We would normally plan this out of hours – However as this was causing issues to customers the decision was taken to perform this immediately.

This affected all services directly connected or routed through this switch

The switch reboot has now completed and service should be restored to all locations and services.

If there is still an ongoing issue for any services please report them to support in the normal way

We apologise for the issues this may have caused

 

OUTAGE: scheduled maintenance DSL overnight 10th April 2019

As part of our commitment to service improvement, we would like to inform you that we will be performing planned critical maintenance as part of capacity upgrades on our core network

Maintenance window start: Wednesday 10th April 2019 23:00 GMT

Maintenance window end: Thursday 11th April 2019 06:00 GMT

Expected impact duration: The work will impact some of our DSL subscriber customers, and they will see an intermittent service within the maintenance, with a short period of down time. Service should be restored automatically but it may require a router re-boot. If after this action, service hasn’t been restored, please raise a support case in the normal manner.

OUTAGE: scheduled maintenance broadband 16th/17th Feb 2019

We will be upgrading some of our core routers that terminate broadband sessions during the course of the weekend – each will be down for less than 30 seconds for an important firmware upgrade. These will be staggered over the two days. Most affected customers will reconnect within seconds automatically. If you have any issues, please reboot your router and if that fails for any reason, please call into or email support in the normal way. This will only affect ADSL, FTTC & FTTP lines.  

CLOSED: Portal diagnostics

UPDATE: the supplier code issues have been worked around and all tests are now available again.

—————————–

We are aware of issues working with the APIs from the suppliers that’s affecting the ability to run diagnostics via the Merula portal (https://adslreports.merula.net); we are working on this now and will advise here as soon as this is resolved. Apologies for this loss of service.

OUTAGE: overnight works

RESOLVED: the fibre break was fixed at 8:53pm last night and all services are back to normal. This ticket is now closed.

UPDATE 15:50 — our supplier advises “in relation to the issue identified in Cambridge area regarding loss of service. We are still working hard to resolve these issues as a priority.

Due to heavy traffic in the area, this is impacting our ability to get into the pit location necessary to move services onto spare fibres. Traffic management is required to safely carry out the work and this cannot be implemented until 20:00 this evening due to local authority restrictions.

From 20:00 onwards we can commence repairs so we should see services begin to come back online overnight this evening.

Please accept our apologies at this time as we are treating this matter with the utmost urgency.”

UPDATE: 14:45pm — because of the amount of splicing work needed and the traffic management problems at this location, the ERT has been pushed out to 20.45 today.

UPDATE: 13:00pm — Traffic management is only permitted until 1530Hrs but traffic lights and barriers are on site ready to go. We’ve not been advised of a new ERT as yet.

UPDATE 12:30pm — work continues on-site at the node. Raised for traffic management. Has ERT for 15 minutes time but engineer still working and waiting for response from traffic management team.

Next update due at 1300Hrs.

UPDATE 10:31am — at 1008Hrs an engineer arrived on site. The overnight change was moving to new cable but there appears to be a fibre break. No ERT at present.

UPDATE 10:11am — this is now being treated as a high priority fault as completion has overrun and classified as a MSO due to the affect on Merula (and other) customers through this node.

UPDATE: we are escalating this for a substantive update as the works have again over-run their new completion time.

Emergency unscheduled works by one of our main backhaul suppliers have overrun and mean that one of our 1Gb links to London from the data centre here in Huntingdon is hard down. Traffic is being routed via one of our backhaul links but as this is slower you may see some slowness on some traffic until this work is completed.

We have been told that the work was due to finish at 6am but this has now been pushed out to 8am.  We will continue to update here. Our apologies if this affects your connection(s).

PLANNED WORK: 1/2/18 UPDATE

Further to the outage yesterday and the corrective work by the carrier we will attempt to bring the link that failed back into service this evening. If all works to plan there may be a brief network disruption while the network reconverges. If the link fails again we will remove it from the network and continue work with our supplier.

The link was brought on line approx 12:30am this morning – However the previous issue re-occurred. This has been raised to the supplier who believe this is resolved this correctly now, We will plan a further maintenance shortly to bring the link back into service. In the mean time there is no reduction to service – but there is a slight reduction in resilience to our Manchester PoP

OUTAGE: Network drop 31/1/18

During planned work deemed low risk by a supplier they managed to inject a loop into one of our links. This caused a significant level of packet loss into our network at approx 22:50 on 31st January.
The link was removed from use and service was resumed albeit with reduced resilience.  The issue has been reported to the carrier who have identified a potential problem and resolved this.

EMERGENCY CONTACTS

The main support number is 0845 330 0666 (geographical 01480 355566)

There’s a second, fallback (geographical) number: 01480 411616. All numbers ring directly at our support centre, manned 24hrs x 365 days a year.

We'd also suggest that all customers subscribe to our mailing list (link above); status messages and updates will be delivered by email.

Subscribe