Incident Details

Resolved [29/01/2022 14:47]

We are now seeing service restored to the remaining NNI. The majority of associated circuits are showing as up, however if you have any issues please reboot the NTU and any associated supplyed routers before raising a fault. We apologise for this prolonged and unexpected outage today.

Update [29/01/2022 11:06]

Splicing works are continuing. As soon as our remaining NNI are connected the remaining circuits will be back online. Apologies for the continued delays caused by Virgin.

Update [29/01/2022 05:46]

2 of the affected 4 NNI have now been restored and associated circuits are reconnected. Virgin warn that full service may not be restored until 11am.

Update [29/01/2022 05:36]

2 of the affected 4 NNI have now been restored and associated circuits are reconnected. Virgin warn that full service may not be restored until 11am.

Investigating [29/01/2022 04:59]

-- MAJOR OUTAGE - Virgin Media LD8 Data Centre --

Virgin Media are aware of a major outage impacting multiple NNI at our LD8 POP. The outage is affecting multiple ISP and will be causing the outage of thousands of circuits. The latest received reads:

<-- snip -->

Major Incident Management were engaged by BusOps at 01:13 to report multiple services as unavailable in the Poplar area. This has been confirmed to be caused by an ongoing High Change C01257982 (WO26506 Poplar Site Exit TXD653 Fibre Joint Swap Out SB351). The impacted customers were not notified of the Change. As the fibre has been cut a rollback of this Change is not possible. MIM has requested the Operations Centre Manager (OCM) confirm whether any of the impacted customers could be prioritised as the Change is scheduled to continue until 11:00 today. MIM remains engaged with all Support Teams at this time.

Plan - OCM to confirm whether any of the impacted customers can be prioritised during splicing. Restoration and Root Cause

<-- snip -->

We are told that a team are on-site and splicing work is set to begin within the next 30 minutes. Further details will be shared once received. Apologies for the disruption caused.