Telehouse Maintenance Tonight (06/07/2016) [ COMPLETED]

Following on from the outage last week we are seeing a issue with some IPv6 traffic being sent via our thn-gw1 router. To fix this we need to restart one of the routing processes. While this is low risk we will perform this out of hours tonight at approx 10pm.

We are not expecting any issues from this – however there is a small risk of a network blip if we have to reboot the router rather than restart the stuck process.

We will update this ticket as the work progresses

This work is now completed and the issue should be resolved – if anyone has any issues please contact support in the normal way

 

Network Blip 10/07/2016

Due to a crash in one of our core routers in Telehouse North a few minutes ago this rebooted causing a very brief loss of service to come customers as the backup router took over the routing.

Customers directly connected would have seen a slightly longer outage. All routers are back up and running and full service should be restored. If you have a broadband service with us and are seeing an issue PLEASE try to reboot as this should solve any remaining issues.

We apologise if not all calls where answered during the issue- the team where busy answering calls and ensuring the issue was resolved as quickly as possible

Notification of Planned Work: 12th August Telehouse North – Virgin Media Business

We need to let you know about some maintenance work that will be taking place on our network. We will be upgrading one of our network back-haul links into the VMB node at our London, Telehouse North site.

Planned Work Start (Local UK): 12/08/2016 00:01 Planned Work End (Local UK): 12/08/2016 06:01

Impact: There will be a loss of service at some point during the timeframe stated above whilst we complete the maintenance work. You will see several interruptions to your service during the 2 hour outage window. We have planned and believe that all circuits will automatically return to service after the work is completed

Only those circuits supplied via VMB will be affected. Please email into support in the normal manner if you have any questions.

[CLOSED] OUTAGE: VOIP service

[update 17:40]

We believe that all connections have now been restored. If you still have VOIP problems, please raise a ticket in the usual manner.

We apologise for this down-time and are actively investigating alternatives and fail-over solutions as this is the second such outage in a few months.

[update: 16:00]

We understand that services are slowly being restored. This may take some time so please bear with us — we’re in the hands of the external supplier here. Further updates will follow as we get them through from the supplier network.

[update 14:52]

The supplier advises that the incident currently affecting their network is down to a Denial of service attack. They are now working with transit vendors to stop the attack and restore the network. Further updates will follow.

[update at 13:49]

Please be advised that the supplier is continuing to investigate the cause of this issue and apologise to all our customers who are affected by this. Further updates will follow.

We’re aware of issue affecting our core supplier which has dropped all of our VOIP circuits including our own PBX. There is no time to fix yet but we’ll update this ticket as we know more.

UPDATE: upgrading Virgin Media fibre into 5 Avro Court 23rd & 24th March

In preparation for additional fibre capacity being brought into the building tomorrow evening — with work commencing on both dates at 6pm — tonight we will force the traffic over our other two backup supplier back-haul channels to ensure that when the Virgin circuit is swapped over to the new fibre connections tomorrow, no down-time is seen.

This may result in some very temporary route instability and slight increase in routing times seen as the work starts & ends tonight and tomorrow. This should resolve automatically.

As always, if you have any concerns, please contact support in the usual manner.

INFORMATION: transit backhaul maintenance work 5th April

Date of work: 5.04.16
Start time: 04:00 GMT
End time: 05:00 GMT
Place of work: London Telehouse North
Work description: Software upgrade on transit supplier core routers

We believe there will be no impact but there could be some very temporary route instability and slight increase in routing times seen as the work starts & ends. This should resolve automatically.

Planned works: 30th March, London transit interconnect

One of our interconnects in THN may see temporary up-ticks in latency and longer routing choices because of the work planned below. This will not affect connectivity and will revert to normal after the work window has closed.

Service Affected: 3rd party Transit, London Bearer

Start of maintenance: 30/03/2016 23:00
End of maintenance: 31/03/2016 03:00
Duration of work: Up to 15 minutes within the outage window

Classification of works: Standard
Description of works: Essential Third Party maintenance

Justification: Planned works to perform JunOS software upgrade.

INFORMATION: broadband connections

Tonight we are performing an urgent security update on a number of our core routers that handle FTTC and ADSL connections.

This will manifest as a brief loss of connectivity, possibly twice during the evening but only for a period of a couple of seconds as the sessions are dropped and fail-over to one of our other routers.

The connection should automatically re-establish itself. If after a power-down of your router for 10 minutes and a re-boot you still aren’t able to re-connect, please email into support@merula.net for assistance.

INFORMATION: Network enhancements between THN & HEX 10pm 14th March 2016

At 10pm tonight, we are bringing an additional new resilient link on line between these two sites and as part of this process you should be aware that there is a small risk of some transient disruption to routing tables. This will not affect connectivity and any affected routes should re-converge quickly.

UPDATE: This work has been completed – the new link is now being monitored

UPDATE: Posting email via Merula servers

[Update 15th March] This work has been completed
[Update 14th March] As of tonight, the main outgoing server will be switched to the new site; anyone using an SSL connections needs to ensure they have changed their outbound server name to port.merula.net
[Update] The new post.merula.net server is up and running – If you use SSL or wish to test sending email please try changing to this server name and ensure you can send email OK. Once we are happy we will update the DNS so that post.merula.co.uk also points to the new server.
We are in the process of bringing into service additional upgraded hardware for our outbound email servers (post.merula.net and post.merula.co.uk). At the same time, we have taken the opportunity to upgraded the security certificate.
A side effect of the certificate change on the mail server will require anyone using SSL in their email client to change the Outbound server name from post.merula.co.uk to post.merula.net
The new servers will be made live over this weekend to allow people time to make the necessary changes.
As always, if you have any concerns, please raise a ticket to support@merula.net.