Edit: Hopefully none of you will see this in the future
Originally posted March 13, 2018 on AIXchange
During a recent HMC upgrade, a buddy of mine had a problem. While you’re unlikely to find yourself in this situation, if you ever do, you’ll be glad you read this.
He was trying to create a new VIO server in enhanced mode, and it kept failing. After opening a PMR and sending in logs, he got this back from IBM Support.
I’ve gone through your logs and found that you have hit a known issue. The problem actually occurred when you upgraded the HMC. During the upgrade some users and groups were recreated with new UIDs and GIDs, rather than being restored, so files in /data that existed prior to the upgrade are orphaned, and the new versions of those users do not have full access to them. The restore upgrade data never actually completed.
PMC0000E: An unexpected Throwable was caught.
Throwable=”PmcJobException” with message “com.ibm.pmc.rest.templates.library.api.LibraryAccessException: Couldn’t create directory /data/pmc/templates/systemtemplate/deploydraft”
Cause=”PmcJobException” with message “com.ibm.pmc.rest.templates.library.api.LibraryAccessException: Couldn’t create directory /data/pmc/templates/systemtemplate/deploydraft”
com.ibm.pmc.jaxb.api.server.jobs.PmcJobException: com.ibm.pmc.rest.templates.library.api.LibraryAccessException: Couldn’t create directory /data/pmc/templates/systemtemplate/deploydraft
/tmp/ls_all.out:
6291473 4 drwxr-xr-x 2 503 504 4096 Nov 7 16:53 /data/pmc/templates/systemtemplate
/data/pmc/templates/systemtemplate
You have a few options to recover from this condition.
First, if you upgraded from 850 to 870 and you saved the SaveUpgradeData off to USB, you can scratch install the HMC to 850 and then restore upgrade data after the scratch install at 850. You would then need to install PTF MH01730 and THEN upgrade to 870. If you no longer have that data, you won’t be able to restore any upgrade data. If you upgraded from 860, you’ll have to scratch install.
The second option would be to scratch install 870 without any restore of upgrade data. We have a doc that tells you the information you will need to document for the new install.
Scratch Installation of Version 8 HMC from Recovery DVD
Items to Document Prior to Performing Scratch Installation of the HMC
The third option is not supported, but many customers have had good luck with this method. You can run the following commands as root to correct the ownership/permissions issues. This is not guaranteed but many have had good luck with it. If any problems arise during or after these steps, you will have to scratch install the HMC.
find /data -uid 501 -exec chown ccfw {} +
find /data -uid 502 -exec chown soliddb {} +
find /data -uid 503 -exec chown wlp {} +
find /data -gid 501 -exec chgrp ccfw {} +
find /data -gid 503 -exec chgrp soliddb {} +
find /data -gid 504 -exec chgrp wlp {} +
find /data -gid 508 -exec chgrp hmc {} +
You will need to be root to try the workaround. Here are the needed passwords to gain root access.
ssh in as hscpe user
enter hscpe password
run PESH <serial number>
enter password of the day
run su –
enter your root password
Obviously for that third method to work, you’d need to get the hscpe password from IBM Support, but it is something to keep in mind.
Again, you’ll probably never run into this, but if you do, this at least gives you some ideas. In any event, open up a PMR and let support guide you.