A Troubleshooting Follow-up

Edit: More fun with zsnap.

Originally posted September 8, 2015 on AIXchange

Last week I wrote about the zsnap command and how it can be used to collect information and troubleshoot data for AIX, PowerHA or VIO server. Here’s how to use zsnap with PowerHA SystemMirror:

The following procedures are for data collection, not for problem diagnosis. Gathering this information before calling IBM support can aid in problem determination and save time resolving Problem Management Records (PMRs).

Using zsnap for PowerHA SystemMirror
Run # zsnap –HACMP
This zsnap command gathers PowerHA data and creates the testcase file in one step. If you already have a PMR number, see the example below.

Data
The zsnap command for PowerHA SystemMirror gathers the same information as snap at this time. The data include:

* Data from both nodes
* CAA data (PowerHA 7.1 and up)
* RSCT information (PowerHA 6.1 and lower)
* AIX information: bootinfo, lslpp, emgr, lsdev disk data, lspv lsvg, lsfs, mount, df, lscfg, lsattr on fibre channel adapter, process table, env data
* Network information: netstat -in, netstat -rn, netstat -v, netstat -m, lsdev adapter and interface data, tty, lsattr on network adapters ODM data for both PowerHA and AIX
* Error report
* Configuration files: clhosts, clinfo.rc, harc.net, netmon.cf, rhosts, clip_config,environment, inetd.conf, limits, profile, resolv.conf, snmpd.log, snmpdv3.log, filesystems, inittab, netprobe.log, rc.net, services, snmpd.peers, syslog.conf, clvg_config, hosts, ipHarvest.log, netsvc.conf, rc.nfs, snmpd.conf and snmpdv3.conf, ifrestrict
* AHAFS data
* PowerHA logs: autoclstrcfgmonitor.out, autoverify.log, cell temp log, clverify, clavan.log, cluster.log, clcomd.log, clcomddiag.log,clconfigassist.log, hacmp.out clstrmgr.debug, clstrmgr.debug.long clevents, clevmgrdevents, clinfo.log, clutils.log, clver_CA_daemon_invoke_client.log, clver_debug.log, cspoc.log, dhcpsa.log, dnssa.log, domino_server.log, emuhacmp.out, hacmprd_run_rcovcmd.debug, application monitor logs, smart assistant logs, smit.log, migration.log
* PowerHA data: hostname information, cllsif information, cluster state data, cluster daemon data, resource group information, cluster topology information

Example
See zsnap usage for all available options.
# zsnap –HACMP –pmr 12345,123,123
The example gathers the appropriate data and creates a testcase file with the IBM standard naming convention for quicker processing. You will be prompted to send the file to IBM using the FTP protocol. If you don’t have a PMR number, omit the –pmr flag to build the testcase file.
You can also run the zsnap command from the AIX SMIT menus.

Using snap for PowerHA SystemMirror
The snap command is the standard AIX tool that gathers data and stores that information in /tmp/ibmsupt/. The snap command does not gather the following additional PowerHA related information.

Data
See zsnap Data section above for the data collected by the snap command.

Sample snap procedure for PowerHA
See snap usage for all available options.

Follow these steps to gather the PowerHA data.
1. Run the snap -r command to remove all previously gathered data on all of the nodes in the cluster.
2. Gather the additional information and put it in /tmp/ibmsupt/testcase. You may need to recreate the testcase directory.
3. Run # snap -e on just one node.
4. Rename the testcase file to adhere to IBM testcase file naming conventions, and then send the file to IBM.

Although IBM Support will guide you through the process of collecting and sending data, it’s best to be proactive. You’ll generally resolve the issue more quickly if you do your own troubleshooting.