Our Network monitoring has detected a potential power issue at our co-location location in Manchester. Our routers and switches at this location are currently unreachable
We are working with our provider to resolve the cause of this issue an update a soon as we have one. Currently there is no service outage – However this is used as a backup / failover location and as such some services should be considered ‘at risk’ until the service is restored
UPDATE: the supplier code issues have been worked around and all tests are now available again.
—————————–
We are aware of issues working with the APIs from the suppliers that’s affecting the ability to run diagnostics via the Merula portal (https://adslreports.merula.net); we are working on this now and will advise here as soon as this is resolved. Apologies for this loss of service.
RESOLVED: the fibre break was fixed at 8:53pm last night and all services are back to normal. This ticket is now closed.
UPDATE 15:50 — our supplier advises “in relation to the issue identified in Cambridge area regarding loss of service. We are still working hard to resolve these issues as a priority.
Due to heavy traffic in the area, this is impacting our ability to get into the pit location necessary to move services onto spare fibres. Traffic management is required to safely carry out the work and this cannot be implemented until 20:00 this evening due to local authority restrictions.
From 20:00 onwards we can commence repairs so we should see services begin to come back online overnight this evening.
Please accept our apologies at this time as we are treating this matter with the utmost urgency.”
UPDATE: 14:45pm — because of the amount of splicing work needed and the traffic management problems at this location, the ERT has been pushed out to 20.45 today.
UPDATE: 13:00pm — Traffic management is only permitted until 1530Hrs but traffic lights and barriers are on site ready to go. We’ve not been advised of a new ERT as yet.
UPDATE 12:30pm — work continues on-site at the node. Raised for traffic management. Has ERT for 15 minutes time but engineer still working and waiting for response from traffic management team.
Next update due at 1300Hrs.
UPDATE 10:31am — at 1008Hrs an engineer arrived on site. The overnight change was moving to new cable but there appears to be a fibre break. No ERT at present.
UPDATE 10:11am — this is now being treated as a high priority fault as completion has overrun and classified as a MSO due to the affect on Merula (and other) customers through this node.
UPDATE: we are escalating this for a substantive update as the works have again over-run their new completion time.
Emergency unscheduled works by one of our main backhaul suppliers have overrun and mean that one of our 1Gb links to London from the data centre here in Huntingdon is hard down. Traffic is being routed via one of our backhaul links but as this is slower you may see some slowness on some traffic until this work is completed.
We have been told that the work was due to finish at 6am but this has now been pushed out to 8am. We will continue to update here. Our apologies if this affects your connection(s).
RESOLVED: 13:14pm
We are not aware of any ongoing issues now and believe that the cause of this problem has been identified and remedial action taken. Once again, apologies to anyone affected this morning.
UPDATE:
We have removed one of the backhaul lines from our network as this appears to be causing routing issues; we are seeing the majority of the affected lines coming back to their normal latency and response times.
We will continue to update here and apologise again for this affecting you at the start of the working week.
We are aware of an as yet unidentified issue affecting large numbers of our circuits leading to slow-downs, poor quality links and high latency. We are working on this now and will update here as soon as we have more information to share. We apologise that this is affecting you on a Monday morning.
UPDATE:
We have seen the services starting to recover and our normal traffic profile is virtually back to normal. Any subscribers still to reconnect may require a router reboot if the issue persists.
The fault is still open with our supplier until the overall service has been restored. Our apologies again to those affected.
+++++++++++++++++++
One of our back-haul providers is aware of an ongoing issue affecting a small section of our lines which is causing either packet loss or intermittent connectivity or sometimes both. NOTE: This isn’t affecting all lines but the following STD codes are those seeing issues through this supplier. We expect an update by 14.30. In the meantime, we apologise if your line is one of those affected.
01171 01173 01179 01200 01214 01282 01372 01483 01485 01512 01513 01514 01515 01517 01518 01519 01527 01553 01604 01628 01905 01932 02010 02011 02030 02031 02032 02033 02034 02035 02070 02071 02072 02073 02074 02075 02076 02077 02078 02079 02080 02081 02082 02083 02084 02085 02086 02087 02088 02089 02311 02380