11/14/2025 Maintenance Notice Reminder

Maintenance Reminder – Scheduled for Today After 10:00 AM EST (UTC-5)

This is a reminder that our previously announced maintenance will be taking place today after 10:00 AM EST (UTC-5). This maintenance window will include multiple critical upgrades to power, network, and compute infrastructure. Please note that all services will be unavailable during this period.

• UPS Maintenance - Replacement of multiple UPS units to improve power reliability and protection. This includes PDU replacements as well, that have been the cause of multiple outages in the past.
• Core Networking Upgrade - Upgrade to a multilink core switch, including installation and configuration of a new core switch to enhance network performance and redundancy. This will improve speeds across all components and increase redundancy. This has contributed to at least 1 major service outage during the period surveyed.
• Ceph Storage Adjustments - Consolidation of multiple Ceph OSDs and retirement of older nodes to improve overall storage efficiency. Nodes to be retired include: PVE7, PVE6, PVE3, and PVE. These nodes are considered EOL and have been migrated from successfully on the software side. These only remain due to the current network configuration. These will be removed from the dual-ring topology today and retired. This has to be conducted during downtime, and will not signifigantly impact downtime today. This will increase performance and remove items no longer needed from production. We have also removed from service prior to this maintenance window multiple drives that have been problematic with either low speeds or high fault rates.
• Compute Node Upgrades - Node PVE9 will receive significant hardware upgrades, including an increase to 256 GiB RAM, new upgraded processor installations, and the addition of three NVMe drives. This will increase performance of our storage array, and allow for more clients to be served on PVE9 concurrently.

The maintenance window is expected to last approximately 4–6 hours, during which our team will be actively monitoring all operations to ensure work is completed efficiently and safely. All services will be fully restored once the upgrades are complete.

You spoke, we listened! Why are these upgrades taking place?

We appreciate your understanding and cooperation as we continue to enhance the stability, performance, and resilience of our infrastructure. These improvements are in direct response to us reaching out, and clients/users indicated their primary focus/concern was stability.  This form of communication was via public surveys conducted via our Discord channel and through individual solicitations/inquiries. All of these improvements are in direct response, as each of these improvements tackles core components that in evaluation each contributed to significant downtime events in the last 11 months. This system refit has been a few months in the works, and we are finally here. Almost all components are "plug and play" outside of network configuration, with as much work completed prior to this time as possible to limit the downtime to as little as possible. 

Original Notice: https://billing.virtnet.bond/announcements/11142025-ups-replacementnetwork-upgrade-planned-maintenance-notice