Sep 23, 2017 | Outages
We have lost power to our rack in Harbour Exchange Square. Our UPS held power for a while but the batteries are now exhausted meaning that services that are provided from Harbour Exchange Square are currently affected – this primarily relates to some of our Leased Lines which are single homed. Most other services have re-routed via alternative Data Centres.
The Data Centre Technicians are working to restore power to the rack asap and we will then expect to see services recover here.
We will update this further as we have the updates.
Note this does not affect services (including leased lines) from other data centres – although there may have been some network instability initially
We are sorry for this issue
UPDATE 14:15 We are starting to see power restored to our rack though there are still some service affected – many are now restored. We are working through these issues and will update this further – However in many cases you should see service restored now
UPDATE 14:22 Equinix (Our Data Centre Supplier in HEX) have just emailed a Incident Update confirming a possible power issue at the facility. We are continuing to see services restore. There are a few remaining services down and we continue to work to resolve these asap. NOTE we have used the opportunity with the power loss to complete the UPS batter replacement – so that there will be no further maintenance on the power within our rack and in the unlikely event of another power fail we now have new batteries in the UPS
UPDATE 15:30 We have restored most services now although it seems the power failure has caused a switch to fail in the rack. All critical services have been moved off of the affected switch and a replacement is being organised to swap out hopefully later this afternoon/evening. There should now be no affected services – However the network should be deemed at risk due to the reduction in redundancy. We will update this once the switch replacement starts
UPDATE 21:10 A replacement switch is now in place and configured in Harbour Exchange square and the remaining services (and resilience) are now restored. There is a need to investigate the power issue further and the data centre may need to change the breaker we are connected to. However this will be a separate planned works and will be announced later. This may be at sort notice BUT will be out of core hours – and will not be today.
We believe service is now fully restored – IF anyone has any ongoing issues please raise them to support via the normal means.
Sep 17, 2017 | Information, Outages, Unplanned downtime, Update
10:23am UPDATE: the supplier reports that the problem has been resolved and we believe that all circuits are now back online. Affected circuits may need to reboot their router to bring their session back on stream.
The following exchanges are affected by this issue since 6.21am this morning.
BT and the supplier engineers are en-route to work on-site. No time to fix yet but we will update here as we hear more.
Exchanges affected include Barrow, Buntingford, Bottisham, Burwell, Cambridge, Crafts Hill, Cheveley, Clare, Comberton, Costessey, Cherry Hinton, Cottenham, Dereham, Downham Market, Derdingham, Ely, Fakenham, Fordham Cambs, Feltwell, Fulbourn, Great Chesterford, Girton,Haddenham, Histon, Holt, Halstead, Harston, Kentford, Kings Lynn, Lakenheath, Littleport, Madingley, Melbourne, Mattishall, Norwich North, Rorston, Science Park, Swaffham, Steeple Mordon, Soham, Sawston, Sutton, South Wootton, Swavesey, Teversham, Thaxted, Cambridge Trunk, Trumpington, Terrington St Clements, Tittleshall, Willingham, Waterbeach, Watlington, Watton, Buckden, Crowland, Doddington, Eye, Friday Bridge, Glinton, Huntingdon, Long Sutton, Moulton Chapel, Newton Wisbech, Parson Drove, Papworth St Agnes, Ramsey Hunts, Sawtry, Somersham, St Ives, St Neots, Sutton Bridge, Upwell, Warboys, Werrington, Whittlesey, Woolley, Westwood, Yaxley, Ashwell, Gamlingay and Potton
We are aware some other exchanges may be impacted
Update – we have just started to see some circuits recover but have no update from the carrier as yet
Jun 23, 2017 | Information, Outages
We are aware that our post.merula.net server was slow / unresponsive between approx 5pm and 8:30pm this evening.
This was due to a customer sending a significant volume of SPAM through the server. We have blocked the affected customer and cleared the Mail Queue on this mail server. Service is now restored.
NOTE that there is a small chance of some outgoing emails being lost while the queue was being cleared – although we believe all legitimate emails have been sent OK.
IF you have sent an email during this time via post.merula.net and have not received a reply you may which to re-send it to ensure it has arrived
We apologies for any issues caused here
Jun 13, 2017 | Information, Outages, Unplanned downtime, Update
RFO 09:40am:
To fix an issue with transient IPv6 and other intermittent routing issues we had seen recently, we were obliged to upgrade the software on one of our core routers. This holds live and backup routes that allow a smooth failover in the case of a single router failing in London. However it now (in an undocumented change from the software supplier) appears with the latest software that the routers set themselves as live on both primary and backup routers – resulting in a routing loop for some IP addresses which had static IP routes originating from this one affected router thus not correctly falling over as previously was the case.
Again, please accept our apologies for this short outage. It shouldn’t have happened.
We are aware of the cause, the problem has now been fixed on this, the one affected router and we have also made sure that all others in the network have been checked and we are confident all are now running properly.
UPDATE 09:27am:
We are aware of the root cause — located at a core switch in our London locations — and are working on bringing this back into service. No ETA yet but we expect this to be resolved shortly. Apologies for the downtime some of you are experiencing.
09:09am We are aware of reports of leased lines down and are investigating. More updates here as we know the cause & ETA to fix.
May 18, 2017 | Information, Outages, Unplanned downtime
We apologise but are aware of a number of leased lines that have dropped because of a hardware fault on a core switch in London.
We will have this restarted within the hour and will be replacing this switch with a new one overnight. More status updates will be posted here.
[update] The initial issue was resolved before 6pm and the replacement hardware was installed in London this evening. This is being monitored but we hope this will resolve the issue once and for all
Apr 4, 2017 | Information, Outages, Planned Work
We are seeing a few ADSL/FTTC lines that we are unable to monitor internally to track performance and transfer speeds. To overcome this we may need to reboot an internal switch that will mean that the PPP sessions on these few lines will momentarily drop. The circuits should all come back on-line automatically, but if you find an issue please re-boot your router to establish the session.
