Friday, February 29, 2008

Working With and Cleaning Out wtmpx

% last

# This example keeps only last 500 records. You might want more on a busy system

% /usr/lib/acct/fwtmp < /var/adm/wtmpx | tail -500 | /usr/lib/acct/fwtmp -ic > /tmp/wtmpx

# Test it
% last -f /tmp/wtmpx

% cat /tmp/wtmpx > /var/adm/wtmpx

Thursday, February 28, 2008

Weekend Down in Flames Revisited

The error from the previous post came back, and all four drives went bad again. This time though, the field engineer replaced the I/O expansion boards and the cable connecting them.

When I went to reboot, it went to book off of dkc0. You might remember that last time, dkc0 had gone bad during the field engineer's fiddling and I had booted off of dkc1 and used volrootmir to remirror dkc0.

Well, the boot didn't go so well. It said it found a valid boot block, but when LSM went to load, it spit out errors about a bad boot track and unmirrored something or other and then went into single user mode. Already running 10 minutes late getting the system back online, I just booted from dkc1 again and it worked fine.

I then removed rz16 (dkc0) from the LSM mirror and then readded it and remirrored. The volrootmir command went something like this, this time:

# volrootmir -a rz16

INFO: The '-a' option was specified for the /usr/sbin/volrootmir command,
however there are partitions on dsk1 that are not
encapsulated and therefore can not be mirrored.


INFO: The '-a' option was specified for the /usr/sbin/volrootmir command,
however the volume on dsk11 is not in the rootdg disk
group and will not be mirrored.


INFO: The '-a' option was specified for the /usr/sbin/volrootmir command,
however the volume on dsk14 is not in the rootdg disk
group and will not be mirrored.


INFO: The '-a' option was specified for the /usr/sbin/volrootmir command,
however the volume on dsk15 is not in the rootdg disk
group and will not be mirrored.


INFO: The '-a' option was specified for the /usr/sbin/volrootmir command,
however the volume on dsk10 is not in the rootdg disk
group and will not be mirrored.

Mirroring system disk dsk1 to disk rz16.
Mirroring rootvol to rz16a.
Mirroring swapvol to rz16b.
Mirroring vol-rz16g to rz16g.

Hmmm. It appears to have worked fine, though there's that note about partitions not being encapsulated. I assume it's referring to the empty partitions or the LSMsimp partition.

I'm left with a sinking feeling though. If the dkc1 disk I booted from goes bad, will I be able to boot from dkc0, or will I face a world of hurt and effort trying to boot from CD and restore from tape? I'd better schedule some downtime soon to try and boot from dkc0 and file a call with HP if it doesn't work. Just four more months with this server before we retire it!

In the meantime, the users are sucking up disk space faster than a "Stand by Me" leech sucks balls. I grabbed a couple of the unused 4.3GB drives, added them to the LSM config in voldiskadm as prod09 and prod10, then did:

# volassist make prodvol-09 8373900s prod09
# volassist -g prod mirror prodvol-09 prod10
# addvol /dev/vol/prod/prodvol-09 gfs_prod

I have about 8GB more disk space left. I hope it's enough to last four months.