11/14/2025 Maintenance Completion Report

Maintenance Update – All Tasks Successfully Completed

We are pleased to announce that all components of the scheduled maintenance originally planned for 11/14/2025 have been successfully completed.

During the process, we experienced a delay related to MTU configuration issues on the new core network switch, which directly impacted Ceph communication. This required additional time to diagnose and correct before we could safely proceed. The issue was fully resolved, and proper MTU alignment across all Ceph paths was verified before concluding the maintenance window. This fix was essential to completing today’s work and ensuring long-term stability of the storage cluster.

Summary of Completed Work

UPS & Power Infrastructure Upgrades
All UPS replacements and PDU upgrades have been completed as planned. These changes improve power reliability, address previously identified failure points, and significantly reduce the risk of power-related service interruptions.

Core Networking Upgrade
The new multilink core switch has been installed, configured, and placed into production. Despite the temporary delay due to MTU mismatches affecting Ceph traffic, all network upgrades have been finalized and are now fully operational. Network performance and redundancy improvements are already noticeable across the environment.

Ceph Storage Adjustments
All planned Ceph operations—including the removal of legacy OSD nodes (PVE7, PVE6, PVE3, and PVE)—were successfully completed. These EOL nodes have been retired from the dual-ring topology, and the cluster has been validated for performance and reliability. Prior problematic drives remain decommissioned as planned.

Future Storage Node Plan
Although PVE3 has been retired during this maintenance window, it is planned to be re-implemented in the future as a dedicated storage node to replace PVE2. This transition will occur once new 2.5" drives have been ordered, delivered, and prepared for deployment.

Compute Node Status (PVE9)
PVE9 remains in its current configuration at this time. Hardware expansion plans originally outlined will be scheduled for a later maintenance window.

Completion & Restoration

All maintenance activities have now been completed, and all services have been fully restored. Our team monitored each stage closely to ensure a safe and efficient process, addressing issues promptly as they arose.

Thank You

This maintenance window was the result of months of planning and guided directly by client feedback emphasizing improved stability and reliability. Your input continues to shape our priorities, and today’s work directly resolves several sources of major outages from the past year.

Thank you for your patience and understanding throughout this important upgrade process.