Edit: Some links no longer work.
Originally posted May 8, 2012 on AIXchange
As I’ve noted before, I love passing along tips and tricks. And I love hearing IT horror stories. A little of both this week:
* First, here’s an email I got from someone who changed the xfer_size option on his machine. In his words:
“I found this blog post a day or two too late. I had tried some AIX fibre card tuning on my Domino servers, which consist of two physical and two VIO NPIV virtual adapters.
“I meant to only change the real cards to 200000 for the xfer_size option, but I had changed the virtual adapters as well, and rebooted. The LPAR hung at LED code 554, and I had to mount OS disk in maintenance mode to mess around with it. This allowed me to undo my change and rmdev the OS disk and paths, to get the LPAR back.
“FYI in case anyone tries this in the future, hopefully they will learn from my mistakes. I am not sure if this is an XIV or an NPIV issue, but I would advise people to not mess with the NPIV xfer_size settings, especially for their root disk.”
* I found this item on a mailing list (from Phil L.) and feel it’s worth sharing:
“[VIOS 2.2.1.3 introduced] clustering… which is controlled by System Director. System Director automatically starts snmpd, so even if you have it disabled with the viosecure commands it will still start via System Director. The workaround:
dirsnmpd (Systems Director) is started from:
/opt/ibm/icc/cimom/bin/startdirsnmpd
To inhibit dirsnmpd at bootup:
Edit: startdirsnmpd script
Comment-out:
# /usr/bin/startsrc -s snmpd > /dev/null 2>&1
“IBM Support is considering modification of the viosecure rules.”
* I saw this in a recent e-mail:
“We have two VIO servers that need to be updated from version 1.4.1.2-FB-9.2 to version 2.2.0.13-FP24 SP03. We did update the two VIO servers in our second data center (so it was a non-production environment). The problem was that we did it incorrectly.
“We put in the migration CD and ran ‘updateios; instead of booting off the CD and running ‘migrateios.’
“We had to rebuild the whole environment from scratch. We definitely want to avoid that in the production environment!”
Me now. All of you have a test lab to try things out first, right?
“I did have a backup of the VIO servers (created via the command ‘ioscli backupios –file … -mksysb,’ run as padmin) but we were unable to recover from that backup.
“We worked with IBM Support and they still were not able to recover from our backup so we had to do a fresh install of the VIO server software and rebuild the environment from scratch, and then I had to
recover each LPAR (12) from their mksysb’s. It was a 40-hour weekend!
Me again. All of you know your backups are good and you’ve tested your recovery process, right?
“In fairness, a lot of that time was actually waiting for CDs to spin. I think if the migration had been done correctly there would only have been a handful of commands that actually needed to be run.”