Here are some commands related to diagnosing problems with FC disks/LUNs and showing multpath status if using Sun's stmsboot/MPXIO/Multipathing software:
# luxadm display /dev/rdsk/c0t0d0s2
# mpathadm show lu /dev/rdsk/c4t600...dd11d0
# mpathadm list mpath-support
# mpathadm list lu /dev/rdsk/c4t600...dd11d0
# mpathadm show mpath-support libmpscsi-vhci.so
# mpathadm show initiator-port 2101...4f93
# fcinfo hba-port
# fcinfo remote-port -l -s -p 2101...4f93
I found many useful when having to check status of and/or compare the paths.
Thursday, May 6, 2010
Solaris 10 SMTP Server Not Running After Patch
About a month ago, I installed over one hundred patches on a Solaris 10 box, including patch 142436-03. Just today I discovered that the server was refusing incoming SMTP connections. (It gets incoming mail only infrequently, obviously.)
I tried to connect to port 25 from another server. No dice. I tried this from the local server:
# mconnect localhost (worked!)
# mconnect actual_hostname (didn't work)
From a Solaris 10 Discussion forum post at http://72.5.124.102/thread.jspa?threadID=5233087, I discovered that the patch must've turned on the local_only configuration by default in an effort to help out with security. I think it would be nice if a patch left things the way they were and maybe prompted you to change it. Oh well. Here's the fix:
# svccfg -s sendmail listprop | grep local_only (to verify it's set to true)
# svccfg -s sendmail setprop config/local_only = false
# svcadm refresh sendmail
# svcadm restart sendmail
Voila!
I tried to connect to port 25 from another server. No dice. I tried this from the local server:
# mconnect localhost (worked!)
# mconnect actual_hostname (didn't work)
From a Solaris 10 Discussion forum post at http://72.5.124.102/thread.jspa?threadID=5233087, I discovered that the patch must've turned on the local_only configuration by default in an effort to help out with security. I think it would be nice if a patch left things the way they were and maybe prompted you to change it. Oh well. Here's the fix:
# svccfg -s sendmail listprop | grep local_only (to verify it's set to true)
# svccfg -s sendmail setprop config/local_only = false
# svcadm refresh sendmail
# svcadm restart sendmail
Voila!
Thursday, April 29, 2010
Cleaning up Solaris 10 Device Tree when LUNs Removed
The following was copied from Symantec here: http://sfdoccentral.symantec.com/sf/5.0MP3/solaris/html/vxvm_admin/ch02s24s03.htm but I added some notes because their instructions were not correct (at least on my system) in some spots.
To clean up the device tree after you remove LUNs
1.
The removed devices show up as drive not available (or drive type unknown) in the output of the format command:
413. c3t5006048ACAFE4A7Cd252
/pci@1d,700000/SUNW,qlc@1,1/fp@0,0/ssd@w5006048acafe4a7c,fc
2.
After the LUNs are unmapped using Array management or the command line, Solaris also displays the devices as either
unusable or failing (or maybe unknown just like all the devices - make sure you have the right ones!).
bash-3.00# cfgadm -al -o show_SCSI_LUN
[...]
c2::5006048acafe4a73,256 disk connected configured unusable
c3::5006048acafe4a7c,255 disk connected configured unusable
[...]
3.
If the removed LUNs show up as failing, you need to force a LIP on the HBA. This operation probes the
targets again, so that the device shows up as unusable. Unless the device shows up as unusable, it cannot be
removed from the device tree. Do a long listing of the rdsk directory to see what device to spevify:
luxadm -e forcelip /devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl
4.
To remove the device from the cfgadm database, run the following commands on the HBA:
cfgadm -c unconfigure -o unusable_SCSI_LUN c2::5006048acafe4a73
or this one if not unusable:
cfgadm -c unconfigure -o c3::5006048acafe4a7c
5.
Repeat step 2 to verify that the LUNs have been removed.
6.
Clean up the device tree. The following command removes the /dev/rdsk... links to /devices.
$devfsadm -Cv
To clean up the device tree after you remove LUNs
1.
The removed devices show up as drive not available (or drive type unknown) in the output of the format command:
413. c3t5006048ACAFE4A7Cd252
/pci@1d,700000/SUNW,qlc@1,1/fp@0,0/ssd@w5006048acafe4a7c,fc
2.
After the LUNs are unmapped using Array management or the command line, Solaris also displays the devices as either
unusable or failing (or maybe unknown just like all the devices - make sure you have the right ones!).
bash-3.00# cfgadm -al -o show_SCSI_LUN
[...]
c2::5006048acafe4a73,256 disk connected configured unusable
c3::5006048acafe4a7c,255 disk connected configured unusable
[...]
3.
If the removed LUNs show up as failing, you need to force a LIP on the HBA. This operation probes the
targets again, so that the device shows up as unusable. Unless the device shows up as unusable, it cannot be
removed from the device tree. Do a long listing of the rdsk directory to see what device to spevify:
luxadm -e forcelip /devices/pci@1d,700000/SUNW,qlc@1,1/fp@0,0:devctl
4.
To remove the device from the cfgadm database, run the following commands on the HBA:
cfgadm -c unconfigure -o unusable_SCSI_LUN c2::5006048acafe4a73
or this one if not unusable:
cfgadm -c unconfigure -o c3::5006048acafe4a7c
5.
Repeat step 2 to verify that the LUNs have been removed.
6.
Clean up the device tree. The following command removes the /dev/rdsk... links to /devices.
$devfsadm -Cv
Friday, February 29, 2008
Working With and Cleaning Out wtmpx
% last
# This example keeps only last 500 records. You might want more on a busy system
% /usr/lib/acct/fwtmp < /var/adm/wtmpx | tail -500 | /usr/lib/acct/fwtmp -ic > /tmp/wtmpx
# Test it
% last -f /tmp/wtmpx
% cat /tmp/wtmpx > /var/adm/wtmpx
# This example keeps only last 500 records. You might want more on a busy system
% /usr/lib/acct/fwtmp < /var/adm/wtmpx | tail -500 | /usr/lib/acct/fwtmp -ic > /tmp/wtmpx
# Test it
% last -f /tmp/wtmpx
% cat /tmp/wtmpx > /var/adm/wtmpx
Thursday, February 28, 2008
Weekend Down in Flames Revisited
The error from the previous post came back, and all four drives went bad again. This time though, the field engineer replaced the I/O expansion boards and the cable connecting them.
When I went to reboot, it went to book off of dkc0. You might remember that last time, dkc0 had gone bad during the field engineer's fiddling and I had booted off of dkc1 and used volrootmir to remirror dkc0.
Well, the boot didn't go so well. It said it found a valid boot block, but when LSM went to load, it spit out errors about a bad boot track and unmirrored something or other and then went into single user mode. Already running 10 minutes late getting the system back online, I just booted from dkc1 again and it worked fine.
I then removed rz16 (dkc0) from the LSM mirror and then readded it and remirrored. The volrootmir command went something like this, this time:
# volrootmir -a rz16
INFO: The '-a' option was specified for the /usr/sbin/volrootmir command,
however there are partitions on dsk1 that are not
encapsulated and therefore can not be mirrored.
INFO: The '-a' option was specified for the /usr/sbin/volrootmir command,
however the volume on dsk11 is not in the rootdg disk
group and will not be mirrored.
INFO: The '-a' option was specified for the /usr/sbin/volrootmir command,
however the volume on dsk14 is not in the rootdg disk
group and will not be mirrored.
INFO: The '-a' option was specified for the /usr/sbin/volrootmir command,
however the volume on dsk15 is not in the rootdg disk
group and will not be mirrored.
INFO: The '-a' option was specified for the /usr/sbin/volrootmir command,
however the volume on dsk10 is not in the rootdg disk
group and will not be mirrored.
Mirroring system disk dsk1 to disk rz16.
Mirroring rootvol to rz16a.
Mirroring swapvol to rz16b.
Mirroring vol-rz16g to rz16g.
Hmmm. It appears to have worked fine, though there's that note about partitions not being encapsulated. I assume it's referring to the empty partitions or the LSMsimp partition.
I'm left with a sinking feeling though. If the dkc1 disk I booted from goes bad, will I be able to boot from dkc0, or will I face a world of hurt and effort trying to boot from CD and restore from tape? I'd better schedule some downtime soon to try and boot from dkc0 and file a call with HP if it doesn't work. Just four more months with this server before we retire it!
In the meantime, the users are sucking up disk space faster than a "Stand by Me" leech sucks balls. I grabbed a couple of the unused 4.3GB drives, added them to the LSM config in voldiskadm as prod09 and prod10, then did:
# volassist make prodvol-09 8373900s prod09
# volassist -g prod mirror prodvol-09 prod10
# addvol /dev/vol/prod/prodvol-09 gfs_prod
I have about 8GB more disk space left. I hope it's enough to last four months.
When I went to reboot, it went to book off of dkc0. You might remember that last time, dkc0 had gone bad during the field engineer's fiddling and I had booted off of dkc1 and used volrootmir to remirror dkc0.
Well, the boot didn't go so well. It said it found a valid boot block, but when LSM went to load, it spit out errors about a bad boot track and unmirrored something or other and then went into single user mode. Already running 10 minutes late getting the system back online, I just booted from dkc1 again and it worked fine.
I then removed rz16 (dkc0) from the LSM mirror and then readded it and remirrored. The volrootmir command went something like this, this time:
# volrootmir -a rz16
INFO: The '-a' option was specified for the /usr/sbin/volrootmir command,
however there are partitions on dsk1 that are not
encapsulated and therefore can not be mirrored.
INFO: The '-a' option was specified for the /usr/sbin/volrootmir command,
however the volume on dsk11 is not in the rootdg disk
group and will not be mirrored.
INFO: The '-a' option was specified for the /usr/sbin/volrootmir command,
however the volume on dsk14 is not in the rootdg disk
group and will not be mirrored.
INFO: The '-a' option was specified for the /usr/sbin/volrootmir command,
however the volume on dsk15 is not in the rootdg disk
group and will not be mirrored.
INFO: The '-a' option was specified for the /usr/sbin/volrootmir command,
however the volume on dsk10 is not in the rootdg disk
group and will not be mirrored.
Mirroring system disk dsk1 to disk rz16.
Mirroring rootvol to rz16a.
Mirroring swapvol to rz16b.
Mirroring vol-rz16g to rz16g.
Hmmm. It appears to have worked fine, though there's that note about partitions not being encapsulated. I assume it's referring to the empty partitions or the LSMsimp partition.
I'm left with a sinking feeling though. If the dkc1 disk I booted from goes bad, will I be able to boot from dkc0, or will I face a world of hurt and effort trying to boot from CD and restore from tape? I'd better schedule some downtime soon to try and boot from dkc0 and file a call with HP if it doesn't work. Just four more months with this server before we retire it!
In the meantime, the users are sucking up disk space faster than a "Stand by Me" leech sucks balls. I grabbed a couple of the unused 4.3GB drives, added them to the LSM config in voldiskadm as prod09 and prod10, then did:
# volassist make prodvol-09 8373900s prod09
# volassist -g prod mirror prodvol-09 prod10
# addvol /dev/vol/prod/prodvol-09 gfs_prod
I have about 8GB more disk space left. I hope it's enough to last four months.
Monday, January 7, 2008
Eudora no longer <<Dominant>>
Late last week, I began to get this error from Eudora on my Mac:
"error while performing unknown task for <<Dominant>>"
My sponsored version of Eudora was trying every two minutes to contact adserver.eudora.com to fetch an updated advertisement. The problem is, Qualcomm no longer develops nor supports Eudora, and apparently turned off the adserver server. Without a response from the server, the Eudora client would pop up an error every couple minutes.
As a quick-n-dirty patch, I put an entry in my Mac's /etc/hosts file for "adserver.eudora.com" with an IP number that belongs to one of our local web servers and restarted Eudora. I haven't received the error since then. The access log on the web server shows that my Mac is requesting:
"POST /adjoin/playlists HTTP/1.0" 404 285
The web server is obviously replying with a 404 error, but at least Eudora doesn't timeout now. Not the best fix, but the best for now.
"error while performing unknown task for <<Dominant>>"
My sponsored version of Eudora was trying every two minutes to contact adserver.eudora.com to fetch an updated advertisement. The problem is, Qualcomm no longer develops nor supports Eudora, and apparently turned off the adserver server. Without a response from the server, the Eudora client would pop up an error every couple minutes.
As a quick-n-dirty patch, I put an entry in my Mac's /etc/hosts file for "adserver.eudora.com" with an IP number that belongs to one of our local web servers and restarted Eudora. I haven't received the error since then. The access log on the web server shows that my Mac is requesting:
"POST /adjoin/playlists HTTP/1.0" 404 285
The web server is obviously replying with a 404 error, but at least Eudora doesn't timeout now. Not the best fix, but the best for now.
Wednesday, January 2, 2008
Back to that Pesky Virtual Frame Buffer
Sigh. While my procedures with the Solaris virtual frame buffer (see previous posts) has usually been working okay, I had to modify the kill command to make it definitely exclude the grep process even though it should have excluded it anyway - very weird:
/usr/bin/kill `ps -ef | grep Xsun | grep -v grep | grep ":5" | awk '{print $2}'`
The user reports that they're occasionally getting this error in their logs:
fstat: Bad file number
(failed to stat vfb)
stat: No such file or directory
(failed to stat vfb)
fstat: Bad file number
(failed to stat vfb)
stat: No such file or directory
(failed to stat vfb)
X connection to localhost:5.0 broken (explicit kill or server shutdown).
This is due to the VFB running in a local Solaris zone. According to a post I found at http://forum.java.sun.com/thread.jspa?threadID=5233796&tstart=120 one can add /dev/winlock to the local zone this way to solve those errors:
"Steps to add the pseudo device '/dev/winlock' to the local zone:
As a superuser in the global zone,
1. Add the '/dev/winlock' pseudo device to the local zone:
global# zonecfg -z
zonecfg:zonename> add device
zonecfg:zonename:device> set match=/dev/winlock
zonecfg:zonename:device> end
zonecfg:zonename> exit
With this, the local zone will have access to '/dev/winlock' device.
2. Reboot the local zone
global# zoneadm -z reboot
Once the local zone is rebooted, '/dev/winlock' will now be available in the local zone."
I've made the change in the zone configuration so it will take effect the next time I reboot. I can see it now - I'll reboot, the instructions above will have been wrong, the zone won't come up, and it'll take me a half hour to remember what changes I made... hopefully I've made enough notes that I'll remember it.
/usr/bin/kill `ps -ef | grep Xsun | grep -v grep | grep ":5" | awk '{print $2}'`
The user reports that they're occasionally getting this error in their logs:
fstat: Bad file number
(failed to stat vfb)
stat: No such file or directory
(failed to stat vfb)
fstat: Bad file number
(failed to stat vfb)
stat: No such file or directory
(failed to stat vfb)
X connection to localhost:5.0 broken (explicit kill or server shutdown).
This is due to the VFB running in a local Solaris zone. According to a post I found at http://forum.java.sun.com/thread.jspa?threadID=5233796&tstart=120 one can add /dev/winlock to the local zone this way to solve those errors:
"Steps to add the pseudo device '/dev/winlock' to the local zone:
As a superuser in the global zone,
1. Add the '/dev/winlock' pseudo device to the local zone:
global# zonecfg -z
zonecfg:zonename> add device
zonecfg:zonename:device> set match=/dev/winlock
zonecfg:zonename:device> end
zonecfg:zonename> exit
With this, the local zone will have access to '/dev/winlock' device.
2. Reboot the local zone
global# zoneadm -z
Once the local zone is rebooted, '/dev/winlock' will now be available in the local zone."
I've made the change in the zone configuration so it will take effect the next time I reboot. I can see it now - I'll reboot, the instructions above will have been wrong, the zone won't come up, and it'll take me a half hour to remember what changes I made... hopefully I've made enough notes that I'll remember it.
Subscribe to:
Posts (Atom)