Orchestrating Rolling OSD Restarts on Proxmox VE

Orchestrating Rolling OSD Restarts on Proxmox VE

When managing Ceph storage clusters on Proxmox Virtual Environment (PVE), applying configuration updates, clearing memory bloat, or recovering from transient drive timeouts often requires restarting Object Storage Daemons (OSDs).

Restarting all OSDs simultaneously causes cluster instability, I/O blocking, and cascading placement group (PG) degradation. The following Bash one-liner performs an automated, safe rolling restart across all local OSD daemons on a single host node:

systemctl reset-failed 'ceph-osd@*.service'; for id in $(systemctl list-units --type=service --state=running --no-legend | awk '{print $1}' | grep -E '^ceph-osd@[0-9]+\.service$' | grep -o '[0-9]\+'); do echo "Restarting local OSD.$id..."; systemctl restart ceph-osd@$id; until ceph daemon osd.$id status 2>/dev/null | grep -q '"state": "active"'; do echo "Waiting for OSD.$id..."; sleep 2; done; echo "OSD.$id is back active."; done

Anatomical Breakdown

1. Pre-Flight State Cleanup

Bash

systemctl reset-failed 'ceph-osd@*.service';

• The Problem: If an OSD crashed earlier or exceeded restart rate-limits, systemd marks its unit as failed. Under default systemd policies, a unit trapped in a failed state may refuse automatic restarts or skew automated status reporting.
• The Fix: reset-failed clears the error counters, exit states, and freeze flags for all templated ceph-osd@ service instances before running the loop.

2. Local-Only Daemon Discovery

Bash

for id in $(systemctl list-units --type=service --state=running --no-legend | awk '{print $1}' | grep -E '^ceph-osd@[0-9]+\.service$' | grep -o '[0-9]\+'); do

Targeting the correct daemons requires isolating only the OSDs assigned to and running on the active node.

• systemctl list-units --type=service --state=running --no-legend: Queries the kernel-level systemd daemon table in memory for currently running services, eliminating table headers and trailers.
• awk '{print $1}': Extracts the primary unit identifier (e.g., [email protected]).
• grep -E '^ceph-osd@[0-9]+\.service$': Strict regex matching ensures non-daemon Ceph services (such as ceph-mon.target or ceph-crash.service) are filtered out.
• grep -o '[0-9]\+': Strips all prefix and suffix text, extracting only the integer ID (4).

Why not parse directories or Ceph CLI tools?

• Examining /var/lib/ceph/osd/ can catch stale mountpoints, lingering configuration directories from dead drives, or symlinks remaining after OSD redeployment.
• Querying ceph osd ls returns all cluster OSDs across every physical host, not just the local Proxmox node.
• Reading systemd's live run table guarantees zero false positives.

3. Service Cycling and Verification

Bash

systemctl restart ceph-osd@$id;

Restarts the targeted daemon instance. Ceph flushes write queues, terminates the process, and spawns a fresh binary instance reading the updated configuration on disk.

4. Admin Socket Health Polling

Bash

until ceph daemon osd.$id status 2>/dev/null | grep -q '"state": "active"'; do echo "Waiting for OSD.$id..."; sleep 2; done

The critical flaw in many automated restart scripts is relying on basic systemd exit statuses or generic ceph status parsing:

• systemctl is-active only checks whether the Linux process exists; it does not know whether Ceph has initialized its RocksDB metadata engine or established peering connections with the monitor mesh.
• ceph osd is-up output syntax has varied significantly across Ceph releases (Quincy, Reef, and newer versions).
• Parsing ceph -s monitor output during a restart introduces network latency and monitor propagation lag.

The Admin Socket Advantage

The command queries the local UNIX domain socket directly at /var/run/ceph/ceph-osd.<id>.asok:

Bash

ceph daemon osd.$id status

This bypasses the Ceph monitor network completely and queries the running daemon process memory. The daemon returns a JSON payload detailing its internal lifecycle states:

JSON

{
    "cluster_fsid": "...",
    "osd_fsid": "...",
    "whoami": 4,
    "state": "active"
}

The loop polls every 2 seconds until the internal state transitions from booting or pre-publish directly to "state": "active". Once confirmed, the script safely advances to the next local OSD.

Operational Safeguards

Before running automated OSD operations in production environments:

• Verify Clean Cluster Health: Ensure the cluster is not already backfilling or degraded (ceph -s).
• Optional Flag Locks: In clusters under heavy client load, consider setting ceph osd set noout prior to running batch restarts to prevent the cluster from initiating unnecessary data rebalancing if a disk takes longer than mon_osd_down_out_interval to initialize. Unset the flag (ceph osd unset noout) immediately after the loop completes.