After-Action Report: PVE12 Hypervisor Reboot

Affected Node: PVE12, PVE9, PVE4, PVE12, PVE6, and PVE3
Duration of Downtime: Approximately 5–10 minutes for the planned reboot
Total Impact Window: ~2 hours of intermittent instability

Summary:
During the planned maintenance reboot of the PVE12 hypervisor, approximately two hours of instability were observed across the virtualization environment. This maintenance was part of routine updates to ensure continued stability and performance.

Incident Details:
Prior to and during the reboot, NIC issues on PVE9 caused RBD-related problems in the Ceph cluster. These problems affected multiple nodes, including PVE9, PVE4, PVE12, PVE6, and PVE3, and contributed to approximately two hours of intermittent instability.

Resolution and Mitigation:

• The NIC-related RBD issues have now been fully resolved.
• Reboots of the affected nodes restored full Ceph functionality.
• Redundant cables for the PVE12 Ceph network were implemented to increase resilience in case of future network issues.
• Preventive measures have been put in place to ensure these issues do not recur.

Impact:

• Virtual machines on PVE12 and other affected nodes experienced temporary instability but remained largely operational.
• The planned reboot itself resulted in approximately 5–10 minutes of expected downtime.

Conclusion:
The maintenance was completed successfully, all services resumed normal operation, and infrastructure resiliency has been improved to prevent recurrence of similar issues.