The cluster downtime was avoided, thanks to some helpful advice from Dell. Our cluster uses a pair of PowerVault 220S enclosures which are configured in a RAID 5 array. When two of the drives in each enclosure went into predictive failure mode, we needed to replace them to ensure the integrity of our data. Since we’re running Rocks for the cluster os, the dell openmanage tools that would allow us to do the hotswap while the cluster was running weren’t available.
I didn’t want to install the openmanage software since it seemed to have a large number of modules and really looked like it might be work (which I try to avoid). A dell rep I talked to recommended DELLmgr instead. He told me to ignore the other rpms, and install only the following: Dellmgr-5.25-0.i386.rpm . It installs one file, dellmgr.bin, that talks to the PERC controller card and gives you an interface very similar to the one used in the card’s BIOS, no restart required. I was able to fail the faulty drives and do the rebuild without having to alter the cluster’s running state at all.
It’s a shame that dell no longer supports it and hasn’t released a version for the new controller cards.