A page at 2:46 Saturday morning woke me up. Our Lawson server, an aging Digital Alpha 4100 running Tru64 Unix 5.1B was complaining. Right away I knew that my list of chores I had planned for the weekend was going to take a hit.
A volume managed in LSM (Logical Storage Manager) had lost a plex and one of the secondary swap spaces was running unmirrored. It looked like a relatively simple disk failure, so I headed back to bed.
In the morning, I went to the office to gather up some paperwork, then drove to the computer room. The alarm in the disk cabinet was screaming like a banshee, and it was festooned with a number of amber lights. The cabinet contains two sets of redundant HSZ70 scsi controllers. The top set is connected to an expansion unit on the back of the cabinet. There are about 70 disks in the cabinet. Four of them, all on channel 1 off the expansion unit were blinking amber. Yikes!
I checked the CLI interface (Command Line Interface Interface) and could see that two raidsets were running reduced, the third disk was the one that was the swap mirror, and the fourth bad disk isn't used. Since everything was mirrored or raided, the system was still up and functioning. I filed a call with HP support at 1-800-354-9000 (1-800-DIAL-A-PRAYER). I tried for a while to convince the phone support tech that there was probably something wrong with the entire channel on the expansion unit, but the controller didn't have any errors on it, so he eventually sent the call to the local field tech and asked them to bring over four new drives. Uh huh. Fortunately, I know the field tech is better than that, and when he called, I explained the situation.
Later in the evening, the tech arrived at the computer room (I had since gone back home) and he unplugged the expansion unit, blew on it, and put it back in. That brought the disks back online. However, it also managed to take down all the disks on channel one on the front of the cabinet too, including the mirror of the root disk that I'd booted from. He rebuilt all the disks from the hardware side, but it left a mighty mess to clean up on the software side.
With the boot disk down and "replaced", that means a shutdown and boot from the other root mirror to free it up so it can be rebuilt.
1) Disassociate and remove the bad plexes from the root volume
volplex -o rm dis rootvol-01
volplex -o rm dis swapvol-01
volplex -o rm dis vol-rz16g-01
2) Remove the disks from the diskgroup
voldg rmdisk rz16a
voldg rmdisk rz16b
voldg rmdisk rz16f
voldg rmdisk rz16g
3) Remove disks from LSM:
voldisk rm rz16a
voldisk rm rz16b
voldisk rm rz16f
voldisk rm rz16g
4) Physically replace the disks using the HSZ commands (I didn't have to replace mine since it wasn't the disks that failed)
5) Label the new disk
disklabel -wr rz16
If that doesn't work, try:
disklabel -z rz16
disklabel -wr rz16
For me, trying to label the disk didn't work at all because it was the boot disk and the OS still had a partition open and claimed. I had to "shutdown -h now" and reboot from the >>> console prompt using the address of the mirrored (good) root drive. On my system a show bootdef_dev showed that the default boot disk was dkc0..., so I used "boot dkc1" to boot from the mirror.
When the system rebooted, I was then able to continue with the disklabel:
disklabel -wr rz16
6) Mirror the root drive
volrootmir -a rz16
That command will build a mirror on the disk specified.
After all that, I tried to remirror the additional swap volume, but ran into a roadblock.
The disks were set up in LSM when the system was installed using Tru64 4.0x. I had since upgraded to Tru64 5.1B. LSM used to use a private region of 1024, but the new version uses a private region of 4096. When I tried to add the new disk using voldiskadm, and then mirror it, it said there wasn't enough space to complete the mirror:
# volassist mirror swapvol02 swap01
lsm:volassist: ERROR: Cannot allocate space to mirror 8376988 block volume
I then tried to write a disklabel from the good mirror onto the disk before adding it with voldiskadm (dsk13 is the good disk, dsk12 is the new one):
disklabel -r dsk13 > dsk13.lab
disklabel -R dsk12 dsk13.lab
I then added the disk through voldiskadm and chose not to initialize it. But then I got this error:
# volassist mirror swapvol02 swap01
lsm:volplex: ERROR: Volume swapvol02, plex swapvol02-01, block 0: Plex write:
Error: Write failure
lsm:volplex: ERROR: sd swap01-01 in plex swapvol02-01 failed during attach
lsm:volplex: ERROR: changing plex swapvol02-01:
Record not in disk group
lsm:volplex: ERROR: Attempting to cleanup after failure ...
lsm:volassist: ERROR: Could not attach new mirror(s) to volume swapvol02
lsm:volassist: WARNING: Object swapvol02-01: Unexpectedly removed from the configuration
Ouch. I turned back to Google and found two tech forum posts from people who'd encountered the same thing, but no one had answered them.
I called HP support and they sent me these three simple little commands:
# voldisksetup -i dsk12 privlen=1024
# voldg adddisk swap01=dsk12
# volassist mirror swapvol02 swap01
Worked like a charm, and the system is back to normal with everything mirrored and raided properly.
However, there's still an amber warning light on the cabinet. Since the controllers aren't reporting any errors, the field tech's best guess is a problem with one of the many fans in the cabinet. He'll be coming by Wednesday with a few new fans to try some replacements to see if he can get that pesky amber light to go away.
