Feb 1, 2018 | Information, Outages, Planned Work
Further to the outage yesterday and the corrective work by the carrier we will attempt to bring the link that failed back into service this evening. If all works to plan there may be a brief network disruption while the network reconverges. If the link fails again we will remove it from the network and continue work with our supplier.
The link was brought on line approx 12:30am this morning – However the previous issue re-occurred. This has been raised to the supplier who believe this is resolved this correctly now, We will plan a further maintenance shortly to bring the link back into service. In the mean time there is no reduction to service – but there is a slight reduction in resilience to our Manchester PoP
Feb 1, 2018 | Information, Outages, Planned Work, Unplanned downtime
During planned work deemed low risk by a supplier they managed to inject a loop into one of our links. This caused a significant level of packet loss into our network at approx 22:50 on 31st January.
The link was removed from use and service was resumed albeit with reduced resilience. The issue has been reported to the carrier who have identified a potential problem and resolved this.
Jan 15, 2018 | Information, Outages, Unplanned downtime, Update
RESOLVED: 13:14pm
We are not aware of any ongoing issues now and believe that the cause of this problem has been identified and remedial action taken. Once again, apologies to anyone affected this morning.
UPDATE:
We have removed one of the backhaul lines from our network as this appears to be causing routing issues; we are seeing the majority of the affected lines coming back to their normal latency and response times.
We will continue to update here and apologise again for this affecting you at the start of the working week.
We are aware of an as yet unidentified issue affecting large numbers of our circuits leading to slow-downs, poor quality links and high latency. We are working on this now and will update here as soon as we have more information to share. We apologise that this is affecting you on a Monday morning.
Dec 4, 2017 | Information, Update
In the event that the phone system here (and the primary support number which routes via here) is unavailable for whatever reason as happened today as our VOIP system failed as part of the outage, there’s a second, fallback (geographical) number: 01480 411616, which rings directly at our support centre. this should be used.
We also suggest that all customers subscribe to our mailing list ensuring that copies of these status messages and updates are delivered to you by email or alternatively if you prefer, to your favourite RSS reader.
If you have any questions please email support@merula.net.
Dec 4, 2017 | Information, Outages, Update
UPDATE2: initial indications on further analysis are that the DDoD that was targeted at the network core which triggered one of our core routers to cease routing packets correctly. However, it did not shut-down the BGP protocols as part of this with the result that network traffic was black-holed for a large number of destinations for a period of around 30 minutes. Service was restored after the routing processes were re-started in sequence. We will continue to investigate further and update here but we believe that all services are now returning to normal. Some routers may have a stale session and will require a reboot to bring them back on-line.
UPDATE: we appear to have been the subject of a wide-spread DDoS attack (the source of which we’re currently still investigating). This caused two of our core routers to become unresponsive and this adversely affected DNS and routing outside our network. We have mitigated the attack here and all of the network is coming back on-line now. Please accept our apologies for this outage — we are aware that it affected a large section of our user-base.
We are aware of an issue affecting large sections of our network centred in London. We are working urgently to fix this and will update here as work progresses.
Nov 30, 2017 | Information
Please note that the offices in Huntingdon will be closed between 5PM 22nd Dec and 9AM January 2nd 2018.
Over this period, urgent telephone support calls will still be answered when calling 0845 330 0666. Email support tickets will be handled intermittently, so if the need is urgent, please call the support line who will escalate if appropriate under the SLA.
Our supplier portals will also be “frozen” so we won’t be processing any new orders during this period.
We wish you all a restful & happy Christmas and New Year.
