OUTAGE: scheduled maintenance broadband 22/1/19
We will be upgrading some of our core routers that terminate broadband sessions later this evening – each will be down for less than 30 seconds for a firmware upgrade. These will be staggered over the evening. Most affected customers will reconnect within seconds automatically. if you have any issues please reboot your router and if that fails for any reason, please call into or email support in the normal way.
This only affects ADSL, FTTC & FTTP lines.
15:42
UPDATE: Avro Court – Power at Risk [resolved]
20/12/18 15:00 UPDATE
The new UPS system has now been installed and commissioned. We now have a full N+1 resilient UPS system in our Avro Court Data Centre. Power is no longer at risk.
19/12/18 10.32am UPDATE
The USP manufacturer has been on-site yesterday and today. After discussions with them, we have decided to take this opportunity to upgrade the current UPS to a later model, giving us increased cover and reliability. This is due to arrive on-site at some point today. Installing, commissioning and testing will however likely roll into tomorrow.
The power supply to the data centre continues to be at risk until this replacement work is completed.
18/12/18 12:10pm UPDATE:
The engineer has sourced the necessary spares and is again en-route to site. We expect that this work will take the rest of the afternoon with the necessary tests being run and, assuming all goes to plan, the UPS will be brought back on-line during the early evening.
The power supply will be at risk until this is completed.
UPDATE:
The UPS is still being worked on. Whilst we expect this to be completed today, power supply to the data centre should still be considered at risk until this is complete.
We had a planned service today of our UPS, by specialist engineers. This has uncovered a problem that has caused our main UPS to not come online fully. Currently services in the Data Centre are working fine but the power should be considered at risk until the UPS is fully restored to service.
The Specialist engineers are currently arranging the parts required and once here they will be swapped out. We will update this post as we have more details
Switch issue – Telehouse West [16/11/18] COMPLETED
We are seeing an issue with one of our Juniper switches in Telehouse West which affects IP addresses directly hosted on the switch. A health check on the swotch shows no issues but routing is impacted.
Currently we have worked round the issue and it is not customer impacting – However we would like to resolve this fully.
As such we will have an engineer on site Friday evening to check and resolve this. We have a spare switch to swap out with if this is a hardware issue
We will update this ticket as the work progresses
Most customers will not notice any issue other than a minor routing change. There are 3-4 directly connected customers who have been notified directly
[update]
This work completed at approx midnight on Friday. The switches have been replaced and the new 10G switches are installed. There was a unexpected issue with Broadband connections terminated in Telehouse East which we are investigating to ensure it does not re-occur
Issues- Manchester
Our Network monitoring has detected a potential power issue at our co-location location in Manchester. Our routers and switches at this location are currently unreachable
We are working with our provider to resolve the cause of this issue an update a soon as we have one. Currently there is no service outage – However this is used as a backup / failover location and as such some services should be considered ‘at risk’ until the service is restored
ADSL Planned Work 18/06/18 – 19/06/18
Overnight on the 18th June – 19th June the link to one of our Wholesale providers is being worked on due to a switch firmware upgrade and reload.
We have dual links to the provider and the work will only affect one line at a time.
During the work (between 10pm on 18th and 6am on the 19th you may see your connection drop as they upgrade and reboot their switches
You should see your connection re-connect after a few seconds on our other link to them.
If your router does not re-connect automatically please turn your router off for 20 minutes to clear any stale session and then re-connect. If this fails – please call or email support in the normal way. We will update this ticket as we receive updates
UPS Maintanance – Avro Court – 18/06/18
As part of our contract with our specialist contractors the UPS in Avro court is having a regular service on 18/08/18.
This involves checking the UPS is functioning correctly, batter health etc
This should not cause any interruption to service but as work is being undertaken here there is always a very small risk.
Any work that is deemed to have a potential of interruption will take place after 5pm. Our own engineers will be on site throughout to monitor the work
We will update this as the work completes
Avro Court Upgrades
Following from the upgrades in London and Manchester the final upgrade will be to our Huntingdon Data Centre. We have a number of planned changes over the coming week.
1) Individual Rack Upgrades
A number of our hosting racks have older cisco switches in them – we are planning on upgrading them over this weekend / early over the coming week. We expect this will cause a number of short breaks to service to individual servers. We will have the new rack in place in each rack before and then simply move the patch cable for each server to the new rack. Once all servers are moved to the new switch – the old switch will be removed.
2) Power upgrades
In preparation for the new core switch units we will be adding additional power to our core racks. This should not cause any issues – However our specialist engineers will be working near the main power distribution board as such this should be considered at risk. All work will be carried out outside normal working hours
3) Core Router and Switch replacement
Once the prep work has been completed we will be replacing the core routers and switches with much faster and newer models. During the replacement process this will mean there are short periods of downtime as services and moved to the new switches and routers. We will create a separate ticket for this work as soon as we have a definate date – however we wanted to provide some notice – we anticipate this being later in the week or over the weekend – it will be outside standard working hours.
We will update this ticket as these progress
Manchester Upgrade 16/06 – 17/06 – completed
Continuing the recent network upgrades on our network the core routers and switches in Manchester are due to be upgraded over the weekend to our new Juniper core.
During the upgrade process there will be breaks in access to some of our backup transit connections in Manchester and a number of leased lines connected in Manchester. All leased line connections in Manchester are backup links so we do not expect and connections to customers to go down – However there will be a lack if resilience while this upgrade is in place. We will upgrade this notification as the work progresses
[Update] This work is completed and all services at Manchester have been updated. There was an unplanned issue routing to a small number of locations during this work but this was identified and resolved. If you are seeing any issues please contact support via the normal routes
CLOSED: Portal diagnostics
UPDATE: the supplier code issues have been worked around and all tests are now available again.
—————————–
We are aware of issues working with the APIs from the suppliers that’s affecting the ability to run diagnostics via the Merula portal (https://adslreports.merula.net); we are working on this now and will advise here as soon as this is resolved. Apologies for this loss of service.