Remote HMC Upgrades

Edit: Some links no longer work. I still love remote upgrades.

Originally posted January 11, 2011 on AIXchange

Anthony English’s recent blog entry about remotely upgrading the HMC struck a chord with me. How many times have you found yourself on a cold raised floor to upgrade a machine? Wouldn’t you rather do that work from a warm office (or, in Anthony’s case, the deck of a ship in Sydney Harbor)?

I had an HMC running V7.3.3 that needed to be upgraded to V7.7.2 in order to support a new POWER7 720 machine. What were my options? There’s upgrading the way we’ve done it for years: I could download the .iso images and burn them to a CD, and then put the CDs into the DVD drive.

Or, I could order the CDs from IBM and not have to download or burn anything.

Or, I could avoid burning physical media or visiting a cold computer room by attempting the upgrade over the network.

Since this was my first time using this method, I started here and followed the instructions:

Download options for HMC network install images. You have three options for acquiring HMC network install image files:

  • Download all files simultaneously via link to Download Director.
  • Download the files individually.
  • Download all files via an anonymous FTP process at a command line.

Note that in all cases you must download (or copy) the image files to a server that accepts FTP requests. You cannot download these files directly to the HMC.

I went ahead and downloaded the files to an FTP server. Then I followed this information:

“You can upgrade the HMC remotely by using network install images rather than using a Recovery DVD. The HMC commands involved include saveupgdata, getupgfiles, chhmc and hmcshutdown. The network install images are linked off of IBM FixCentral.”

I downloaded the files:

Enter these commands, type.

# ftp
ftp> open ftp.software.ibm.com
  Name: anonymous
  Password: ftp
ftp> cd software/server/hmc/network/v7720
ftp> prompt   (turns off interactive mode)
ftp> bin   (ensures binary transfer mode)
ftp> mget *   (downloads all six files simultaneously)
ftp> bye   (exits FTP after all files have been downloaded)”

These files had been successfully downloaded to my FTP server:

# ls -la
total 5068440
drwxr-xr-x   2 root     system          256 Dec 07 13:25 .
drwxr-xr-x   3 root     system          256 Dec 07 10:07 ..
-rw-r–r–   1 root     system      1531846 Dec 07 10:08 bzImage
-rw-r–r–   1 root     system    672645120 Dec 07 11:03 disk1.img
-rw-r–r–   1 root     system   1133975552 Dec 07 12:25 disk2.img
-rw-r–r–   1 root     system    753831936 Dec 07 13:25 disk3.img
-rw-r–r–   1 root     system           78 Dec 07 13:25 hmcnetworkfiles.sum
-rw-r–r–   1 root     system     33049856 Dec 07 13:28 initrd.gz

I logged into my HMC using ssh as hscroot, and the first command ran fine:

>saveupgdata -r disk

However, I ran into a snag with my second command:

> getupgfiles -h ftpserver -u root –passwd passw0rd -d /fixes/HMCV7R7.2.0
Cannot contact server to obtain files.

A Web search netted me this forum and this question, but, alas, no answer.

So I tried rebooting the HMC. That didn’t help.

I verified I could connect to the FTP server from other machines, but I couldn’t figure out why my HMC wouldn’t connect to the FTP server. Instead of spending more time troubleshooting, I decided to download the files directly from IBM onto the HMC. I ran this command:

>getupgfiles -h ftp.software.ibm.com -u anonymous –passwd ftp -d /software/server/hmc/network/v7720

That worked great, although it took longer to download the files than it would have if I were moving files on the local network. To monitor my progress, I ran the script that’s suggested in this post:

while true ; do
date
ls -la /hmcdump
sleep 60
done

Once the download completes, the /hmcdump filesystem gets unmounted. That tells you that the download is finished.

“The getupgfiles operation will mount a filesystem called /hmcdump and copy the install files into the directory then unmount the filesystem. The following commands will set the HMC to boot from the network install images and allow the upgrade to proceed.”

Once the download completed, I ran:

>chhmc -c altdiskboot -s enable –mode upgrade

>hmcshutdown -r -t now

Then I waited a while until the HMC came back from the upgrade. This was actually the one mildly disconcerting bit of the whole operation; there’s no feedback as to whether anything is happening. How do I know if it’s actually working if I don’t see something telling me it’s working? How do I know that the HMC didn’t have a problem on reboot and it’s not just sitting there, frozen, waiting for someone to physically touch the machine?

In my case it took about 20 minutes for the upgrade to complete, and beyond that I had to wait until I could actually login to the HMC GUI (although I could ping the HMC and ssh into the command line). I kept getting an error: “Console not Ready. You cannot log on at this time. The console is still initializing and is not yet ready for users to login. Allow the console to finish initializing and then try to login again.”

After the upgrade, I made sure to load the mandatory eFix MH01235, along with MH01243 and MH01244. This brought the HMC to the current level at the time of writing.

I have to agree with Anthony: Remotely upgrading an HMC is a very painless way to go, though I still wish there was a way to get status information during the actual upgrade process. Just don’t ask me how IBM could provide that information from a machine that’s hundreds of miles away and not on the network while the upgrade is taking place.