This work will upgrade the capacity on one leg of our dark-fibre ring. Whilst it won’t cause any outages, we are at elevated risk during the maintenance window and there may be brief transient routing issues when the fibre is brought back on-stream.
Maintenance Window: 21:00 – 07:00 GMT
Primary Date: 07-Apr 2017
2nd Primary Date: 08-Apr 2017
3rd Primary Date: 09-Apr 2017
Location of Maintenance: London
Reason for Maintenance: supplier will perform fiber maintenance to add capacity
We are seeing BGP issues affecting one of our core routers in HEX, London. This may involve the need to reboot this equipment late this evening. This will affect a few of our leased line customers only.
We don’t expect this to materially affect any of our services but there may be a few instances of slight packet loss or routing transit issues during this window.
Start: 2017-02-23 00:00:00 GMT (UTC + 0)
End: 2017-02-23 04:00:00 GMT (UTC + 0)
This maintenance is scheduled for the 2nd stage of Juniper software and Routing Engine upgrades.
During this maintenance we will replace both 32 bit RE-S-2000 Routing Engines in Edge1,
Edge2 and Edge3 THN (Telehouse North) with 64 bit RE-S-1800×4 Routing Engines. We will also upgrade the Junos version to 15.1F6-S3 from 13.3R5-S4.1.
All members connected to edge1-thn, edge2-thn and edge3-thn shall experience downtime of up to 20 minutes whilst the RE’s are swapped and rebooted.
The following routers are vulnerable as below. New firmware has been released for some of the models and is available from here: http://kb.netgear.com/000036386/CVE-2016-582384
Netgear R7000, firmware version 188.8.131.52_1.1.93 and possibly earlier, R6400, firmware version 184.108.40.206_1.0.11 and possibly earlier, and R8000, firmware version 220.127.116.11_1.1.2 and possibly earlier, contain an arbitrary command injection vulnerability.
By convincing a user to visit a specially crafted web site, a remote, unauthenticated attacker may execute arbitrary commands with root privileges on affected routers. An unauthenticated, LAN-based attacker may do the same by issuing a direct request, e.g. by visiting:
An exploit demonstrating these vulnerabilities has been publicly disclosed.
Netgear’s advisory confirms that the R6200, R6400, R6700, R7000, R7100LG, R7300, R7900, and R8000 are vulnerable, though affected firmware versions are not enumerated. The vendor has indicated that their advisory will be updated as firmware updates are released.
We are seeing some oddness on one of our LNS routers which terminates some of our broadband circuits. We believe this is only affecting our management access and not the service provided over this.
We are working with the vendor to understand this and hopefully resolve this. Assuming this is not customer affecting we may need to reboot this LNS later this evening causing a brief network blip as the sessions re-connect.
IF we find this is affecting service earlier we may have to perform the reboot earlier. We will update this as we know more
Our monitoring started reporting that one of our core routers in Telehouse (thn-gw2) failed just after 19:15 this evening. We have now rebooted this and the router appears to be back and stable. We will investigate the cause of this and update the ticket as needed.
Most customers would not have noticed this (other than some slightly different routing) and possibly a few short periods of packet loss as routes re-converged. A small number of connections are directly connected and may have seen a longer outage as the router was restored.
We believe full service is now resumed – If you are seeing any ongoing issue please contact support in the normal way
We are sorry for any issues caused by this
UPDATE: the supplier has advised that the work was incorrectly dated as tonight — it’s actually due to occur over the night of 22nd/23rd.
We’ve been notified of some urgent works occurring tonight on one of our backhaul supplier network interconnects.
This will only impact a very small number of our users who are routed through this. If after the work is due to complete, you’re still seeing issues, please reboot your router & if that fails to bring you back on-line, then contact support in the normal manner.
Maintenance window start: 22/11/2016 23:00
Maintenance window end: 23/11/2016 06:00
Expected impact duration: 5 minutes outage
Your service will be at risk of additional or extended outages for the duration of the maintenance window.
Divert Service Failures
Details: SIP Diverts are now working as normal and our engineers will closely monitor our VoIP network for any re-occurrences. If you were affected by the issue please retest and if the problem is still present contact our Support Team.
One of our suppliers is aware of a network problem as below. We’ll update here as they troubleshoot this. In the meantime, please accept our apologies if your circuits are affected.
Network Affected: VoIP
Problem Time: Wed 19th Oct 2016 at 9:30 am
Estimate Resolve Time: Wed 19th Oct 2016 at 10:00 am
What Does It Affect: Customers using SIP divert service
Event Overview: Divert Service Failures
Details: Some of our customers may be experiencing problems with our SIP Trunk Divert Service where inbound calls are failing. Our engineers are currently investigating and there will be further updates shortly.
One of our interconnects in Manchester may see temporary up-ticks in latency and longer routing choices because of the work planned below. This will not affect connectivity and will revert to normal after the work window has closed.
Level3 Transit, Manchester Bearer
Start of maintenance:
25/10/2016 – 00:01 GMT
End of maintenance:
25/10/2016 – 03:00 GMT
Duration of work:
Up to one hour within stated window
Impact of work:
Outage / Disruption to services
Classification of works:
Repair / Planned (Third Party)
Description of works:
Level 3 Communications will implement scheduled maintenance to perform upgrade work on core network equipment to free up memory and prevent future service interruptions to customer services.
We are receiving alerts from our UPS in Telehouse. We have therefore scheduled a maintenance to visit the site and either replace the batteries on the UPS or replace the UPS as needed. Depending on the situation there may be an outage for Telehouse North based services late this evening for upto 90 minutes although we will of course keen this to an absolute minimum.
NOTE this UPS is in addition to the building UPS and Generator.
This work will affect most broadband services and some Leased lines.
We are sorry for the short notice but want to resolve the possibility of an issue as quickly as possible and at a time when least people are using the service
We will update this post as the work progresses