This was resolved approx 4pm after the faulty switch was swapped on the supplier network
[Update at 14:46]
The supplier is advising us that most lines are now returning on-stream; this may take a few more minutes as the Radius catches up. Anyone still affected after this time should power-off their router for at least 20 minutes to clear any stale session. Please email into support@merula.net if this fails to bring you back live.
We apologise for the lengthy downtime and are looking at further remedial work with the supplier to ensure that such a failure doesn’t affect us in future.
[Update at 14:33]
Apologies for the lack so far of anything concrete in time-to-fix terms; we are escalating this to senior managers inside the supplier to get this fixed.
[Update at 13:33]
Senior engineers are currently on site working on the faulty hardware.
Further updates will be posted once the work has been completed.
[Update at 12:35]
Supplier update: We’re seeing a partial recovery on the equipment.
We’re aware some circuits are still down, our engineers are looking to replace some of the
hardware in the switch stack.
Further updates will be posted when available.
[Update at 12:10]
The supplier has a new switch on route to the site to be swapped out — they’re expecting this to complete by 1pm. We’ll update as this progresses.
We are aware of a problem affecting one of the interconnect switches on a transit supplier network which means that a number of lines dropped earlier this AM and are still down; they and we are working on getting this switch bypassed and replaced. Currently we have no time-frame for a fix but believe this will not be service affecting for too long.