Nov 15, 2018 | Planned Work
We are seeing an issue with one of our Juniper switches in Telehouse West which affects IP addresses directly hosted on the switch. A health check on the swotch shows no issues but routing is impacted.
Currently we have worked round the issue and it is not customer impacting – However we would like to resolve this fully.
As such we will have an engineer on site Friday evening to check and resolve this. We have a spare switch to swap out with if this is a hardware issue
We will update this ticket as the work progresses
Most customers will not notice any issue other than a minor routing change. There are 3-4 directly connected customers who have been notified directly
[update]
This work completed at approx midnight on Friday. The switches have been replaced and the new 10G switches are installed. There was a unexpected issue with Broadband connections terminated in Telehouse East which we are investigating to ensure it does not re-occur
Oct 11, 2018 | Unplanned downtime
Our Network monitoring has detected a potential power issue at our co-location location in Manchester. Our routers and switches at this location are currently unreachable
We are working with our provider to resolve the cause of this issue an update a soon as we have one. Currently there is no service outage – However this is used as a backup / failover location and as such some services should be considered ‘at risk’ until the service is restored
Jun 16, 2018 | Planned Work
Overnight on the 18th June – 19th June the link to one of our Wholesale providers is being worked on due to a switch firmware upgrade and reload.
We have dual links to the provider and the work will only affect one line at a time.
During the work (between 10pm on 18th and 6am on the 19th you may see your connection drop as they upgrade and reboot their switches
You should see your connection re-connect after a few seconds on our other link to them.
If your router does not re-connect automatically please turn your router off for 20 minutes to clear any stale session and then re-connect. If this fails – please call or email support in the normal way. We will update this ticket as we receive updates
Jun 16, 2018 | Uncategorized
As part of our contract with our specialist contractors the UPS in Avro court is having a regular service on 18/08/18.
This involves checking the UPS is functioning correctly, batter health etc
This should not cause any interruption to service but as work is being undertaken here there is always a very small risk.
Any work that is deemed to have a potential of interruption will take place after 5pm. Our own engineers will be on site throughout to monitor the work
We will update this as the work completes
Jun 16, 2018 | Planned Work
Following from the upgrades in London and Manchester the final upgrade will be to our Huntingdon Data Centre. We have a number of planned changes over the coming week.
1) Individual Rack Upgrades
A number of our hosting racks have older cisco switches in them – we are planning on upgrading them over this weekend / early over the coming week. We expect this will cause a number of short breaks to service to individual servers. We will have the new rack in place in each rack before and then simply move the patch cable for each server to the new rack. Once all servers are moved to the new switch – the old switch will be removed.
2) Power upgrades
In preparation for the new core switch units we will be adding additional power to our core racks. This should not cause any issues – However our specialist engineers will be working near the main power distribution board as such this should be considered at risk. All work will be carried out outside normal working hours
3) Core Router and Switch replacement
Once the prep work has been completed we will be replacing the core routers and switches with much faster and newer models. During the replacement process this will mean there are short periods of downtime as services and moved to the new switches and routers. We will create a separate ticket for this work as soon as we have a definate date – however we wanted to provide some notice – we anticipate this being later in the week or over the weekend – it will be outside standard working hours.
We will update this ticket as these progress
Jun 16, 2018 | Uncategorized
Continuing the recent network upgrades on our network the core routers and switches in Manchester are due to be upgraded over the weekend to our new Juniper core.
During the upgrade process there will be breaks in access to some of our backup transit connections in Manchester and a number of leased lines connected in Manchester. All leased line connections in Manchester are backup links so we do not expect and connections to customers to go down – However there will be a lack if resilience while this upgrade is in place. We will upgrade this notification as the work progresses
[Update] This work is completed and all services at Manchester have been updated. There was an unplanned issue routing to a small number of locations during this work but this was identified and resolved. If you are seeing any issues please contact support via the normal routes
May 23, 2018 | Information, Outages, Unplanned downtime
UPDATE: the supplier code issues have been worked around and all tests are now available again.
—————————–
We are aware of issues working with the APIs from the suppliers that’s affecting the ability to run diagnostics via the Merula portal (https://adslreports.merula.net); we are working on this now and will advise here as soon as this is resolved. Apologies for this loss of service.
Apr 5, 2018 | Information, Outages, Unplanned downtime, Update
RESOLVED: the fibre break was fixed at 8:53pm last night and all services are back to normal. This ticket is now closed.
UPDATE 15:50 — our supplier advises “in relation to the issue identified in Cambridge area regarding loss of service. We are still working hard to resolve these issues as a priority.
Due to heavy traffic in the area, this is impacting our ability to get into the pit location necessary to move services onto spare fibres. Traffic management is required to safely carry out the work and this cannot be implemented until 20:00 this evening due to local authority restrictions.
From 20:00 onwards we can commence repairs so we should see services begin to come back online overnight this evening.
Please accept our apologies at this time as we are treating this matter with the utmost urgency.”
UPDATE: 14:45pm — because of the amount of splicing work needed and the traffic management problems at this location, the ERT has been pushed out to 20.45 today.
UPDATE: 13:00pm — Traffic management is only permitted until 1530Hrs but traffic lights and barriers are on site ready to go. We’ve not been advised of a new ERT as yet.
UPDATE 12:30pm — work continues on-site at the node. Raised for traffic management. Has ERT for 15 minutes time but engineer still working and waiting for response from traffic management team.
Next update due at 1300Hrs.
UPDATE 10:31am — at 1008Hrs an engineer arrived on site. The overnight change was moving to new cable but there appears to be a fibre break. No ERT at present.
UPDATE 10:11am — this is now being treated as a high priority fault as completion has overrun and classified as a MSO due to the affect on Merula (and other) customers through this node.
UPDATE: we are escalating this for a substantive update as the works have again over-run their new completion time.
Emergency unscheduled works by one of our main backhaul suppliers have overrun and mean that one of our 1Gb links to London from the data centre here in Huntingdon is hard down. Traffic is being routed via one of our backhaul links but as this is slower you may see some slowness on some traffic until this work is completed.
We have been told that the work was due to finish at 6am but this has now been pushed out to 8am. We will continue to update here. Our apologies if this affects your connection(s).
Mar 30, 2018 | Uncategorized
Further to the recent posts – we are planning the final Network upgrades in our core network later today. This will decommission the last of our old network in London. This involves the move of our final leased line bearers and the final Broadband Interconnects.
For Broadband Interconnects you may see your session drop – However we have alternative Interconnects to all wholesale providers in our other PoPs so these should re-connect almost instantly on an alternative link.
Leased Lines terminating in Telehouse will have an outage of at maximum 30 minutes while they are moved between the old and new network. In many cases the outages will be much shorter than this
We will update this post as the work progresses
Mar 14, 2018 | Uncategorized
During the next few weeks we will be migrating our core network onto new, faster Juniper router and switches. The new routers and switches are already in place in our data centres and configured. The current work involves moving links in a controlled manner between the old and new network . The majority of this i moving peering and transit connections one by one. This will not cause any outages – just subtle routing changes as the traffic moves away from a transit – the link is moved and the transit (or peering re-appears via the new network.
Once this is completed there will be some brief blips on leased lines and broadband connections as they are moved to the new network – these will be communicated as these are planed – most outages will be less than a couple of minutes in these cases as the cables are moved.
The end result will be a more modern network with far greater capacity for growth
We will update this site as the plans progress
