Sep 17, 2017 | Information, Outages, Unplanned downtime, Update
10:23am UPDATE: the supplier reports that the problem has been resolved and we believe that all circuits are now back online. Affected circuits may need to reboot their router to bring their session back on stream.
The following exchanges are affected by this issue since 6.21am this morning.
BT and the supplier engineers are en-route to work on-site. No time to fix yet but we will update here as we hear more.
Exchanges affected include Barrow, Buntingford, Bottisham, Burwell, Cambridge, Crafts Hill, Cheveley, Clare, Comberton, Costessey, Cherry Hinton, Cottenham, Dereham, Downham Market, Derdingham, Ely, Fakenham, Fordham Cambs, Feltwell, Fulbourn, Great Chesterford, Girton,Haddenham, Histon, Holt, Halstead, Harston, Kentford, Kings Lynn, Lakenheath, Littleport, Madingley, Melbourne, Mattishall, Norwich North, Rorston, Science Park, Swaffham, Steeple Mordon, Soham, Sawston, Sutton, South Wootton, Swavesey, Teversham, Thaxted, Cambridge Trunk, Trumpington, Terrington St Clements, Tittleshall, Willingham, Waterbeach, Watlington, Watton, Buckden, Crowland, Doddington, Eye, Friday Bridge, Glinton, Huntingdon, Long Sutton, Moulton Chapel, Newton Wisbech, Parson Drove, Papworth St Agnes, Ramsey Hunts, Sawtry, Somersham, St Ives, St Neots, Sutton Bridge, Upwell, Warboys, Werrington, Whittlesey, Woolley, Westwood, Yaxley, Ashwell, Gamlingay and Potton
We are aware some other exchanges may be impacted
Update – we have just started to see some circuits recover but have no update from the carrier as yet
Jun 13, 2017 | Information, Outages, Unplanned downtime, Update
RFO 09:40am:
To fix an issue with transient IPv6 and other intermittent routing issues we had seen recently, we were obliged to upgrade the software on one of our core routers. This holds live and backup routes that allow a smooth failover in the case of a single router failing in London. However it now (in an undocumented change from the software supplier) appears with the latest software that the routers set themselves as live on both primary and backup routers – resulting in a routing loop for some IP addresses which had static IP routes originating from this one affected router thus not correctly falling over as previously was the case.
Again, please accept our apologies for this short outage. It shouldn’t have happened.
We are aware of the cause, the problem has now been fixed on this, the one affected router and we have also made sure that all others in the network have been checked and we are confident all are now running properly.
UPDATE 09:27am:
We are aware of the root cause — located at a core switch in our London locations — and are working on bringing this back into service. No ETA yet but we expect this to be resolved shortly. Apologies for the downtime some of you are experiencing.
09:09am We are aware of reports of leased lines down and are investigating. More updates here as we know the cause & ETA to fix.
May 18, 2017 | Information, Outages, Unplanned downtime
We apologise but are aware of a number of leased lines that have dropped because of a hardware fault on a core switch in London.
We will have this restarted within the hour and will be replacing this switch with a new one overnight. More status updates will be posted here.
[update] The initial issue was resolved before 6pm and the replacement hardware was installed in London this evening. This is being monitored but we hope this will resolve the issue once and for all
Oct 19, 2016 | Information, Outages, Unplanned downtime
UPDATE:
Divert Service Failures
Details: SIP Diverts are now working as normal and our engineers will closely monitor our VoIP network for any re-occurrences. If you were affected by the issue please retest and if the problem is still present contact our Support Team.
One of our suppliers is aware of a network problem as below. We’ll update here as they troubleshoot this. In the meantime, please accept our apologies if your circuits are affected.
Network Affected: VoIP
Problem Time: Wed 19th Oct 2016 at 9:30 am
Estimate Resolve Time: Wed 19th Oct 2016 at 10:00 am
What Does It Affect: Customers using SIP divert service
Event Overview: Divert Service Failures
Details: Some of our customers may be experiencing problems with our SIP Trunk Divert Service where inbound calls are failing. Our engineers are currently investigating and there will be further updates shortly.
Jun 7, 2016 | Information, Outages, Unplanned downtime, Update
[update 17:40]
We believe that all connections have now been restored. If you still have VOIP problems, please raise a ticket in the usual manner.
We apologise for this down-time and are actively investigating alternatives and fail-over solutions as this is the second such outage in a few months.
[update: 16:00]
We understand that services are slowly being restored. This may take some time so please bear with us — we’re in the hands of the external supplier here. Further updates will follow as we get them through from the supplier network.
[update 14:52]
The supplier advises that the incident currently affecting their network is down to a Denial of service attack. They are now working with transit vendors to stop the attack and restore the network. Further updates will follow.
[update at 13:49]
Please be advised that the supplier is continuing to investigate the cause of this issue and apologise to all our customers who are affected by this. Further updates will follow.
We’re aware of issue affecting our core supplier which has dropped all of our VOIP circuits including our own PBX. There is no time to fix yet but we’ll update this ticket as we know more.
Nov 6, 2015 | Information, Outages, Unplanned downtime, Update
15:15 The problem has been traced to one of our core routers which ‘hung’ without (as it should have) notifying the automatic monitoring system of problems. This in turn affected routing for some customers connected to our Telehouse data centre. Some other important switches and routers were also impacted as they were unable to see that this router was down and also failed silently.
We have now restored service to the router and are monitoring for any further issues.
This should not have happened but despite our planning it did and we can only apologise for this; we are working on reconfiguring the core layout very shortly to make sure that this can’t cause such cascading problems for our customers again.
13.30 — we believe that we have resolved these service issues. That said, we are monitoring still and looking for the root cause. Again our apologies for this loss of service and updates will continue to be posted here once we’ve had a chance to check logs etc.
UPDATE: we’re working with our link team as this is mainly affecting services out of our London data centres. Apologies for this extended down-time, we’re all working on this problem and will update here as we know more detail.
We’re aware of and are investigating the cause of outages affecting a number of services inc. some leased lines, ethernet circuits and broadband lines. As soon as we know the root cause & likely time to fix, we’ll update there.
