Author: Rob McNelly

Originally published by TechChannel December 5, 2023

Rob McNelly also highlights information on Power10 performance, AIX administration, VIOS to NIM mapping, and shares his fondness for some old interfaces

In November, IBM TechXchange conducted a deep dive on AIX 7.3 TL updates. Featuring AIX product manager Jayen Shah and Carl Burnet, DE for IBM Power, the webinar covered the latest AIX enhancements:

“In this webcast, you will learn:
*How AIX OS feature enhancements provide the capacity, performance, and leading security needed to accelerate business outcomes.
*How to harness Power automation to stay current with the latest technology while keeping data secure and maintaining optimal performance.”

Register for the replay and download the slides.

Having tuned into this webinar, I’ll share some information that stood out to me. First, a reminder that should have been on your radar and that should not be a surprise, AIX 7.1 has reached end of support (slide 2). If you’ve not yet upgraded to AIX 7.2 or 7.3, you should make the move as soon as possible. There is nothing more frustrating than needing IBM’s assistance and being told that you need to either upgrade or purchase (assuming it’s available) extended support.

Updates to the AIX Collection at Ansible Gallery are highlighted (slides 6-7), along with updates to the AIX Toolbox (slide 8). VIOS support is covered on slide 12. Version 3.1.1 is end of support, 3.1.2, 3.1.3 and 3.1.4 remains in support, with VIOS 4.1 currently available.

On that note, I’m already seeing people test the upgrades to 4.1. As always you can get the latest lifecycle information and release notes for 4.1.0.10. Keep in mind that moving to 4.1 is essentially the same as upgrading your base operating system. In other words, it’s more involved than simply installing a VIOS fixpack. As a reminder, VIOS 4.1 runs a stripped-down version of AIX 7.3 under the covers (slide 13), whereas VIOS 3.1 ran AIX 7.2 behind the scenes.

There’s more, including PowerVC updates (slide 16) and PowerSC coverage (starting with slide 18). Slide 26 gets into training options, including a link to this course list.

How to Tackle Device Package Installation

Last week, IBM Support posted an update regarding device package installation warnings. Unexciting as that may sound, this is useful, bookmark-worthy information:

Problem
The cfgmgr command (or cfgdev on VIOS) displays the following warning and fails to discover a new device:

cfgmgr: 0514-621 WARNING: The following device packages are required for
    device support but are not currently installed.

This message is followed by one or more fileset names such as:
devices.fcp.disk
devices.fcp.changer
devices.fcp.tape
devices.sas.changer

Cause
The cfgmgr command displays this message when it discovers a device for which a driver cannot be identified. However, these filesets listed in the message do not exist. They are generic names based on the type of device. For example, devices.fcp.tape means that a fibre-channel attached tape drive was found devices.sas.changer means that a SAS-attached media changer (tape robot) was found.

Read on for IBM’s recommendations on resolving these issues.

Network Connectivity Simplified

A decade or so ago, we had to jump through many virtual hoops in our efforts to display information for network devices (specifically, Cisco switches) connected to AIX LPARs. Fortunately, this process has become much simpler, as IBM’s Chris Gibson explains in his recent TechChannel article:

“AIX administrators can now display information for Cisco network devices (switches) that are directly connected to their AIX logical partition (LPAR). This is made possible by exploiting the Cisco Discovery Protocol (CDP). AIX provides the cdpd daemon which can receive incoming data packets or messages by using CDP and discovering the physically connected Cisco devices.”

The output and information available to us these days is pretty impressive:

cdpctl show port en1

Waiting for CDP advertise (default 60 seconds)……

Device ID: route1-n1.local(AGE19190TKY)

Address: 10.10.10.2

Port ID: Ethernet3/43

Capabilities

: Router Level 3

: Level 2 Switch

Cisco switch OS Version: Cisco Nexus Operating System (NX-OS) Software, Version 9.3(9)

Platform: N9K-C9508

Native VLAN ID: 32

Trusted Bitmap: N/A

AVVID untrusted ports: N/A

Duplex: Full

MTU: 576

System Name: route1-n1

System Object ID: N/A

Management Addresses: 10.1.2.10

CDP record received on dev en1 also stored at /tmp/cdp_record_en1

Again, that’s from the article. Be sure to read the whole thing.

Optimizing Power10 Performance

Chris also highlighted this IBM doc on Power10 performance topics:

Abstract: “This document aims to offer guidance and topics for optimizing performance for IBM Power10 processor-based servers. It should be noted that this document does not cover all the best practices for PowerVM, AIX, or IBM i, and should be used in conjunction with other relevant documentation.”

VIOS to NIM Mapping

I always try to make sure my NIM server is at the right level to support my VIO servers. Here’s an IBM Support document that can help you do just that. Though you’ll see conflicting dates of publication, I believe this information has been recently updated.

On User Interfaces and Facing Reality

Lately I’ve been thinking about interfaces that I deal with, both on the job and outside of it. One of my favorites is—still—the smitty menus. I love that they look almost exactly the same as they always have, and that most of the system commands I know and love continue to function as they have for years.

Of course, this is an exception. Consider the evolution of the HMC interface and GUI. Every now and again I stumble across an old, unpatched HMC version, and it makes me think of things that used to be second nature. (And never mind the old interfaces; I also miss the old and familiar—and the even older—sounds.)

While we may get eased into changes by being given options to stick with the classic GUI or try out an enhanced interface, eventually, inevitably, the old ways of doing things disappear. Running the new interface is our only choice.

This certainly isn’t exclusive to IBM solutions. Recently I updated the system software for my smartwatch. I immediately noticed that the vendor had made some pretty drastic changes. Menu options and shortcuts were moved, buttons that used to do one thing no longer did anything. Years of muscle memory I’d accumulated were useless, as I had to adapt to new ways of doing things.

But it isn’t just me. Years ago, a coworker fell in love with his Palm Pilot. Even as the world moved on to Blackberrys, Androids and iPhones, he’d live by what I call the “eBay and pray” method of system support. If one device gave out, he’d simply purchase another old used one and move his data there. Eventually though it becomes an exercise in futility. Think of the applications you use—the authenticators, the collaboration tools, things that literally will not run on older hardware and operating systems. The world moves on. Either we move with it or get left behind.

Sure, there are legitimate needs to move forward. Getting the latest bug and security fixes that ship with these updates is essential. Still, it’s easy to wonder if tech companies oftentimes make changes simply for the sake of making change, leaving us users to do the workarounds and roll with the punches. Even as someone who’s spent his career in tech, who knows from day-to-day experience how cool all this stuff is and how much more incredible it’s become over time, if I have the option to stick with an old interface rather than move to a new one, I will choose the tried and true whenever I can, and for as long as I can.

Originally published by TechChannel October 31, 2023

Rob McNelly cites recent AIX enhancements and passes along a plethora of tips

In 2020 I wrote about a couple of scripts called ezh and eezh:

“EZH is a script for the [HMC] to provide an alternate, easier to use, command line interface for many common commands, and the goal of the project is to make the HMC command line interface easier to use for day to day administration tasks.

“…Enhanced EZH script is a fork of EZH by Brian Smith for the [HMC] to provide an alternate, simpler command structure which makes the HMC command line interface easier to use for day to day administration tasks. The vision for this project is to enhance and bring EZH commands forward and inline with technologies provided in the newer HMC code base.”

For those of you who use these scripts, you should know that they’ll break the next time you update your HMC. Matthew Opoka, who maintains eezh, gave me this heads up. He explains that, in addition to ezh/eezh, the latest patch will break any HMC script that utilizes sed, since support for the sed command is basically being removed. (These PTFs are intended to address CVE-2023-38280. See the PTF MF71298 and MF71299 readme files for details.)

It’s possible this feature could be reinstated in the HMC code through the IBM Request for Enhancement (RFE) process. If you use sed with the HMC, I suggest you register on IBM’s RFE portal and share your opinions. For the uninitiated, the RFE community is well worth your time and participation, as it gives all of us the opportunity to help shape the future of Power Systems development.

VIOS to NIM Mapping, Recent AIX Enhancements, IBM PowerDraw

These items come courtesy of IBMer Chris Gibson‘s mailing list.

1. This recently updated IBM Support doc covers VIOS to NIM mapping. Check the link to see the master levels needed if you’re using NIM to backup, install, or update a VIOS partition.

2. Here’s an extensive list of recent AIX enhancements, including AIX 7.3 Standard Edition TL 2 and AIX 7 Enterprise Edition 1.10.

3. And here’s a description of a new configuration tool called PowerDraw, which provides an interactive, graphical representation of Power Systems environments:

The product uses the included capture module to collect all the information about the server, PowerVM (VIOS), and partitions from the HMC into a single text capture file… Any modern browser can display and interact with these diagrams, allowing [clients] to easily explore the relationships with mouse over highlighting of connections, and popup boxes that display all the detailed information. It can be useful as the starting point for problem determination, or for easy evaluation of virtualization best practices. The SVG files can be integrated into browser-based dashboards or used as stand-alone local files viewed by a browser. No web server is required.”

Incidentally, Chris’s mailing list is open to all. Email him if you’d like to be subscribed. 

More AIX Problem-Solving Tips

Here are some other things I’ve recently come across.

* This IBM Support doc describes how to use MTU 9000 (jumbo frame) in an environment with AIX LPAR and VIOS.

* Here’s an oldie but a goodie on preparing a Power server for a concurrent SAS or SSD disk replacement in AIX or VIOS:

“It is important to first determine disk array configuration including protection level and then follow appropriate procedure. To perform these procedures AIX root access is required. On VIOS run “oem_setup_env” to switch from user padmin to root.

“If you already established your level of protection, you can go directly to the Procedure.

• Procedure 1: Replacing an AIX System Disk, hdisk JBOD (Just a Bunch Of Disks)
• Procedure 2: Replacing a pdisk that is part of a RAID0 unprotected array with a single pdisk
• Procedure 3: Replacing a pdisk in a RAID5 or RAID10 protected array
• Procedure 4: Replacing a pdisk in a protected RAID array that was previously a Hot Spare
• Procedure 5: Replacing a pdisk in an unprotected RAID0 array with multiple pdisks
• Procedure 6: Replacing a pdisk in a RAID6 array
• Procedure 7: Replacing a hot spare disk that is either failed or has a PFA (Predictive Failure Analysis) indication” 

* Here’s another useful nugget. Use this HMC command to determine the MAC address of the virtual ethernet adapters on each LPAR:

lshwres -r virtualio –rsubtype eth –level lpar -m <managed system>

* And lastly, a troubleshooter from IBM Support. An AIX system rebooted, but getty didn’t start, preventing a console login:

“The most common reason for getty not starting is failure of a preceding line in /etc/inittab. Since that line does not complete, the line which starts getty is not read….

To troubleshoot:

Check the console setting

# lscons vty0

Check the state of vty0

# lsdev -C|grep vty0

vty0 is available

Check process table for getty

# ps -ef|grep getty

There is no output, which means that the getty process isn’t running.

Check to see what process is prohibiting getty from starting or respawning.

# who -p

“Usually, the culprit is the last process in the who -p output. In this example, it is l2, which is called from the /etc/inittab file. Note: l2 is causing the system to not completely read /etc/inittab.”

See how they addressed the issue.

Originally published by TechChannel September 7, 2023

IBM Support Insights provides a wealth of critical information to make admins’ jobs easier

Like many of you, my organization manages servers, switches, storage devices, etc., in multiple data centers located throughout and even outside of the country. With collections of devices this large, naturally, there are many challenges. How do you manage hardware inventories? How do you know when a new device is added to your environment? How do you track serial numbers? How do you know when your vendor’s maintenance contracts are about to expire? Do you have an overall risk assessment of your environment at your fingertips?

Recently I received access to a systems management tool called IBM Support Insights, which IBM describes as a cloud-based service that allows organizations to proactively improve uptime and address security vulnerabilities. It’s available at no charge with select IBM warranty support and maintenance contracts.

A subscription version, IBM Support Insights Pro, becomes generally available September 12, 2023. Pro, which uses watsonx Assistant for interactive application help and navigation, is designed in particular for organizations in industries with a heightened sensitivity to downtime and security breaches. In addition to standard capabilities, the Pro version offers access to views for analyzing security vulnerability and hardware lifecycle risks, recommended OS and firmware levels, and enhanced case analysis (to be released shortly after announce) and up to 12 months of case history.

Snapshots of Inventories, Hardware Issues and More

By logging into Support Insights, I was able to answer many of the questions I posed above. In one sample demo account that was loaded with real-world data, I discovered that nine new assets were added in the past month. I also learned the date they were added, the machine type, the product family, the serial number, and the vendor. By clicking on individual assets, I could access contract and billing information. I could see where each asset is installed, who it was sold to, and any cases that were opened against that asset—all from a single portal running in my web browser.

I could also check the firmware level, view the expiration dates for hardware support, and be informed of pending OS updates. I could view all cases that had been opened with IBM support: the total created, the number closed and the number of days from creation to close for each individual case. I could sort IBM cases by asset name to see how many calls were being logged against specific machines.

This additional information makes it easier to plan actions around inventories. Organizations can quickly learn when and what assets are approaching end of service, which LPARs need to be updated and which machines they’re spending the most time servicing. They’ll eliminate gaps in service that could lead to extended downtime, and they won’t miss patches on any of their machines.

Getting back to my testing, I found a risks and recommendations page that provided an overall risk score for my environment. I could access visual detail—i.e., charts—to examine coverage issues, OS/firmware end of support events, security vulnerabilities (CVEs) and hardware end of support events. I was able to view all my devices that lacked a support contract, along with assets with a looming expiration of coverage (within 30 to 90 days).

Have you ever tried to open a ticket with support, only to learn that your support contract has expired? Or have you been told that support can’t assist you until you update your LPAR, because the issue you’re dealing with was fixed in a service pack issued months earlier that you haven’t installed? With this information close at hand, organizations can avoid these dreaded but all too common experiences.

Under the OS/Firmware dial I could see which machines needed firmware or OS updates. The Hardware dial displayed alerts for all the hardware that was at or approaching end of life. The Asset Analysis tab—a feature available only to Support Insights Pro subscribers—contains views for analyzing security vulnerability and hardware lifecycle risks and enables users to make reasoned decisions about patches, upgrades and replacement planning by providing details about correlating risks and issues.

All of this information is extremely valuable. You’re afforded a snapshot of your hardware issues. You can determine if your staff is spending a lot of time with the vendor trying to resolve these issues. Support Insights offers visibility to your entire data center’s inventory while providing information from various sources, connectors, and vendors to help you navigate potential issues and recommended actions.

From an IBM perspective, the tool utilizes information derived from the IBM Call Home service to get data related to IBM Power and Storage systems, things like firmware levels. The tool can also source data from IBM contracts, allowing simple access to information on machine types and models, contract coverage information, along with warranty information. This can help populate information around lifecycle status of hardware, along with any coverage or warranty issues.

The IBM Technical Support Appliance (TSA) can be used to automatically discover assets and import data into Support Insights. I briefly describe TSA here.

Support Insights provides lifecycle information for the AIX, IBM i and Red Hat operating systems, along with widely used IBM solutions like VIOS and the HMC. In one location you can view the versions you’re running, along with recommended updates. There is also support for non-IBM servers (Dell, HPE, Oracle, VMWare), storage platforms (EMC, HPE, NetApp) and network and security devices (including Cisco, Juniper, F5, Brocade, QLogic, Check Point, Palo Alto and Fortinet). On that note, Support Insights supports the Cisco CSPC Connector, which provides information about OS levels, security vulnerabilities, hardware, software end-of-support and more. Further, the Support Insights ServiceNow CMDB Connector can pull data from ServiceNow to help populate inventory and provides additional data for analysis.  As of this writing, the ServiceNow Connector is in beta.

Some Views From My Testing

Since a picture is worth a thousand words (or so I’ve been told), I’ll share some screenshots.

Here’s a sample of inventory management:

Figure 1. Inventory coverage from a sample customer

Here’s a view of inventory coverage from a sample customer. These views are customizable. I can see current maintenance and support agreements and determine if some have expired or are about to. With this information, I can quickly and easily identify my most at-risk assets and avoid issues that could trigger unplanned downtime.

Figure 2. Machine type/models and coverage status

By drilling down into the asset names, I could see machine type/models and the coverage status. Running this against real data can help you proactively manage renewals, which saves money, time and resources. I don’t have to wonder whether all my devices are covered. I know.

Figure 3. Renewal management

Have I opened cases with vendors? How many? How long has it taken to get these issues resolved? This information can be used to inform business units about vendor relationships and help the organization learn to identify any machines that have abnormal service patterns.

Figure 4. Case metrics dashboard

What risks exist in my environment? Are they related to security or back-level OSes and hardware that is end of support? With prioritized at-risk assets identified, I can plan for patches and move to supported hardware when necessary.

Figure 5. Risk and recommendations dashboard view

Figure 6. Risks and recommendations asset list

These views are from Support Insights Pro. They enable users to understand security vulnerability and hardware lifecycle risks at a deeper level and make better, more informed decisions by correlating risk factors across security, hardware and software end of support, and support coverage.

Figure 7. Risks and recommendations, continued

Providing Visibility

During my training, I came across this summary of the tool:

“IT inventory management requires visibility first and foremost. Without visibility across the IT estate, it is difficult and time-consuming to identify the most critical issues and assess the business impact. IBM Support Insights provides automated multi-vendor infrastructure asset discovery, customized inventory views and exports, and support coverage and change analysis.

“By understanding the overall risk of their IT estate (servers, storage and networking), clients can benefit from taking on-time and preventive actions to improve their overall availability and reduce the risk of security breaches and service disruption.”

An interactive demo is available here. You’ll need an IBM ID to log in. Or contact ibmsi@us.ibm.com for details.

Originally published by TechChannel August 24, 2023

Rob McNelly urges old dogs to learn new tricks and shares additional thoughts about the recently retired Nigel Griffiths

In March, Jaqui Lynch wrote about connecting HMCs to Power10 servers.

If you’re in the process of moving to Power10/updating your HMCs, this is a must-read. If you’d like me to summarize it, I can do so in a pithy phrase: The only constant is change.

The modern HMC has different connectivity methods compared to the original versions that us old dogs relied upon for all of these years. If it’s been some time since you’ve refreshed your Power or HMC hardware, you need to understand the changes that have been implemented. The Power-based HMC running V10 has yet another new interface, and there are new ways to connect the HMC to the network. While most of us have moved from the classic HMC interface to the enhanced version over the past few years (unless you stayed on old, unsupported hardware and code levels, which creates its own set of problems), this latest change is something we must all deal with eventually as we refresh our environments. If your current hardware is, actually, relatively current, that may be a bit down the road yet. Still, it never hurts to think about it in advance.

Jaqui’s article includes a number of useful supporting links. One is an IBM-produced set of videos on installing Power10 servers and connecting them to the HMC. Give them a watch. These short videos are packed with important information.

Similar to the HMC’s transformation, Power10 is a slightly different animal compared to POWER8/9. To provide some idea of the content, here’s a list of individual video titles:

• “A look at the top three issues (HMC Access ID, VMI, and ACFs) seen on POWER10 eBMC systems”
• “SSH with eBMC”
• “eBMC Basic User Functions”
• “Firmware update via eBMC ASMI menu”
• “Configuring a new eBMC POWER10 system without an HMC”
• “Lost Admin Password Recovery”
• “Configuring a new POWER10 eBMC system with a DHCP address provided by the HMC”

Once you’re up and running in your new environment, you’ll find that many of the tasks that you’ve grown accustomed to remain the same. The major changes are mostly confined to the process of setting up your system for the first time. In addition, you’ll find some differences in patching and connecting to the BMC.

Speaking of the HMC

This is an oldie but a goody. You have a server that’s been assigned an IP address from the HMC. How do you determine what that address is? Function 30 is what you need:

“The default TCP/IP addresses for POWER server FSP ethernet ports HMC1 and HMC2 differ depending on the platform and server firmware level. The following table documents the default IP addresses of both ports for both single and redundant FSPs (if installed).

“Note: IBM POWER8, POWER7, and POWER6 servers with firmware Ex340 or later implement zero configuration (zeroconf) networking for the FSP. The default settings remain the same however it implements zero configuration networking to prevent duplicate IP addresses, should multiple servers be plugged into the same network with no DHCP server available. When the FSP is plugged into a network with no active DHCP server, it will locate a unique IP address in the 169.254 range.

“To determine the current IP address of the FSP, use control panel function 30. For further information see IBM support document N1015416 Panel Function 30.”

One other HMC-related item: The next session of the Power Systems Virtual User Group is devoted to HMC and VIOS, with Jaqui Lynch handling the presentation:

“VIO servers are the most critical part of your system setup. If they are not happy, then no client LPAR will be happy. This session provides tips on setting up and maintaining VIO servers including upgrades and patching. Backup and recovery will also be covered. The HMC is a critical component of your environment and has become far more complicated. Time permitting, we will cover the differences between the old Intel HMC and the new POWER 7063 HMCs. We will also cover maintenance for the HMC and BMC and some of the new techniques and options made available with version 10. Tools such as FLRT, FLRTVC, and the HMCScanner will also be discussed.”

While the live event has already taken place, a replay will be uploaded in the near future.

Still At It

Not surprisingly, Nigel is still on social media, extolling the capabilities of IBM Power Systems/AIX.

I refer to him as Nigel, because the first name alone is sufficient. Of course, I mean Nigel Griffiths, who retired from IBM in July. I mentioned it in passing in my previous column, but I planned on saying a bit more once I had time to gather my thoughts.

Nigel will always be a unique presence in our world. Seriously, who else is there in the Power/AIX space who’s instantly recognized on a first-name basis? It sure isn’t Rob.

When I think of Nigel, I always come back to the sheer volume of work he’s produced over the years. His YouTube page features 200+ videos, some going back more than a decade. This is to say nothing of his countless in-person presentations at conferences, or the webinars and lectures he presented, the first looks at hardware, the infographics, the simple to understand demos of new technologies and enhancements.

Then there’s the NMON and NJMON performance monitoring tools. It is a sign of confidence in your skills when you name your application Nigel’s Monitor. It’s a sign of your talent and ingenuity when the AIX world adopts it, and IBM itself recognizes its value and adds it to AIX:

The original nmon version was for the IBM AIX operating system (Release 4.3 and above) and was a freely downloadable binary format only tool from the IBM AIX wiki. Later a version was written for the Linux operating system running on IA-32, x86, x86_64, IBM RS/6000 and POWER processors, mainframe and ARM (including Raspberry Pi). nmon for Linux was released by IBM as open source in July 2009. The code is available from the Sourceforge open-source repository.

“The nmon for AIX code was later bundled in as part of the AIX operating systems. From AIX 5.3 TL09 and AIX 6.1 TL02 onward it was included in the default installation of AIX and fully supported by IBM. The nmon command and the topas command are the same binary but behave differently depending on the command name used.”

Honestly, it’s hard to think of anything I’ve learned over the years that didn’t somehow involve Nigel, either as a presenter or as a technical expert answering questions in the background.

Voids occur when people move on, so I’m very happy to see that Nigel has merely stepped back, rather than walk away completely. Hopefully we’ll continue to hear more, wherever his future adventures take him.

Originally published by TechChannel August 15, 2023

Rob McNelly on the challenges posed by some technological innovations, notes from IBM Support, and a shout-out to the newly retired Nigel Griffiths

I recently went on a wild ride while with family. Thanks to a last-minute cancellation, my sister and niece needed to get from Rochester, New York, to Newark, New Jersey, to catch their connecting flight home to Phoenix.

That flight, from Newark was scheduled to leave around 8 p.m. We were notified of the cancellation around noon, which at least gave us some time. While the airline did offer them a flight from Rochester to Chicago, there was a catch: two more connecting flights would follow, from Chicago to Los Angeles, and then from Los Angeles to Phoenix.

As a seasoned traveler, I knew that itinerary was problematic. A delay at any point, and my sister and niece could be left to wander through airports for a day or longer. It made the most sense to keep that original nonstop flight and find another route to Newark. Since I already had a rental car with a full tank of gas, I decided to channel my inner Elwood Blues and hit the road.

The drive took roughly 11 hours round trip. That’s about as far as driving from Phoenix to Denver, or Phoenix to Salt Lake City, or Phoenix to San Francisco. As I’ve mentioned, I’m good with driving. Last fall I drove across the country and back. Just a few weeks ago I chose to skip the hassles of flying and drive some six hours to a customer site in California. Really, considering the time I’d spend parking at the airport, going through security and waiting at the gate, the time commitment was about the same flying or driving.

But back to the Rochester to Newark run. Adding to the excitement, a thunderstorm rolled through; it literally rained the whole way. Living in the Arizona desert as I do, this isn’t something I’m used to. My rental car was another unfamiliar space. It was a 2023 model with only about 5,000 miles on it. This vehicle had all the modern bells and whistles and tech.

Odd as this may sound coming from me, a technology professional, I prefer older vehicles. They’re reliable, and they’re cheap to register and insure. I drive my cars into the ground before I replace them.

Certainly, the technology in new vehicles is impressive, but it seems each innovation leads to new things that can go wrong. In this case, it was the cruise control. In many new vehicles, the cruise control system utilizes sensors that determine the proximity of your vehicle to the one ahead of you on the highway. If the car in front of you slows down, your cruise control slows you down to maintain safe separation. If that vehicle ahead of you speeds up, so does yours, automatically. This is all fine and good in normal weather conditions, but again, it was raining. And once the rain got really heavy, that triggered an error message on my dash. The sensor stopped working during the downpour, so the cruise control stopped working, and there was no way to override it from what I could tell. So, for a time I actually had to use the gas pedal. That felt like the Dark Ages, or maybe the Flintstones. Anyway, not being able to set it and forget it with the cruise made things a bit more arduous.

I’ve found similar challenges with four-wheel drive. In my old 4×4 Suburban, I just press a button and I’m good to go. Newer vehicles have computerized traction control that are designed to compensate for the driver’s abilities; they do things that you might not expect. This and other automated settings can get you in trouble if you don’t realize they’re active.

Not that there isn’t precedent for that sort of thing. A Windows laptop will update and reboot itself every few weeks. Automatic updates are great, provided everything is working as expected. But what if some application stops working because of issues with a patch? What if you were not expecting a reboot and you lose work that was in progress, or, in the worst-case scenario, what if a bug with the patch causes the OS itself to stop booting? As frequently and fervently and as I advocate for everyone to regularly patch their systems, I wouldn’t want the production systems that I work on to automatically download patches from IBM and reboot without my input.

In any event, the rain eventually let up and the error with the sensors went away, which allowed me to reengage the cruise control. I spent the rest of the trip wistfully thinking of my old car and its simple settings.

News and Tips from IBM Support

1. An important update on regarding Call Home and electronic fix distribution: “Due to technical issues, the May 30/31 change will not become active until Sept 19/20, but leave in your firewall configuration if already configured from our previous guidance.”

2. This information on AIX ulimit and maxuproc may come in handy: “A design change took effect in AIX 7.2 TL5 and AIX 7.3 so that ulimit shows the value set for maxuproc. Since processes per user, or nproc, is not set in /etc/security/limits, the value shown in previous releases of AIX was always unlimited…. The design change ensures that ulimit shows what the actual limits are for this value.”

3. Get an rPerf performance rating: “Run this script of your AIX server to calculate an estimated rPerf rating for that particular logical partition (LPAR), also called virtual machine (VM). The estimate is based on the number CPUs and Machine-Type-Model (MTM) calculated from the official documented rPerf ratings.”

4. I ran into this recently after upgrading VIOS: “Starting in VIOS version 2.2.4.10 and above, the following message of the day (motd) message is displayed upon login after the VIOS is updated to 2.2.4.10 or higher…”

In my case I had to run the rulescfgset command twice for it to work.

Nigel Griffiths Retires from IBM

Did you see that Nigel Griffiths recently retired from IBM? Nigel is truly a unique presence who has contributed so much to the AIX world. I, for one, have cited his deep dives into the technology almost constantly over the years. But on that note, if IBM is looking for someone to carry on with the behind the scenes videos of their brand new servers, I’m certainly available. Seriously though, congrats and best wishes to Nigel.

Originally published by TechChannel June 22, 2023

Rob McNelly experiments with the AI tool to see what it knows about IBM Power Systems and AIX

I assume none of you live under a rock, so you’ve most likely heard about people’s experiences toying around with ChatGPT and other AI solutions. I decided to get in on the fun, so recently I put Google’s bot—it’s called Bard—through its paces.

I wanted to see what Bard knows about IBM Power Systems and AIX. Let’s just say the answers I received were underwhelming.

POWER9 Chips

My first query was a simple one: Can you still get POWER9 chips? The response:

“Yes, you can still get POWER9 chips. IBM is still manufacturing and selling POWER9 chips, and there are a number of other companies that make POWER9-based servers. Some of the companies that sell POWER9 chips include: IBM, Raptor Computing Systems, Suzhou Powercore Technology, Tyan, SGI, Lenovo.”

This isn’t entirely accurate. I know about systems from Raptor, and I’ve occasionally come across articles that reference Suzhou working with modified Power chips in China. Tyan did produce a POWER8 system, but I’m not aware of anything from SGI, which went bankrupt and was bought out in 2009.

I decided I should be more specific. I asked which Lenovo systems use POWER9 chips. Here’s Bard:

“Lenovo offers a variety of systems that use POWER9 chips, including:
ThinkSystem SR650
ThinkSystem SR950
ThinkSystem DX360 M5
ThinkSystem DX750 M5
ThinkSystem SD650
ThinkSystem SD750
ThinkSystem TH240
ThinkSystem TH440
ThinkSystem TH540”

Me again. You likely know, as I do, that ThinkSystems are rack servers that run Intel processors. But plenty of folks, some of whom may even oversee your IT department, wouldn’t know the difference. Taking this information at face value could lead to some very uninformed decisions.

AIX vs. Linux

So, I took a different approach and made my queries more general. I asked about the advantages of AIX over Linux. The response:

“AIX has several advantages over Linux, including:

• Stability: AIX is known for its stability and reliability. It has been around for over 30 years and is used by some of the world’s largest companies.
• Performance: AIX is a high-performance operating system that can be used for a variety of demanding workloads.
• Security: AIX is a secure operating system that is regularly updated with security patches.
• Support: AIX is supported by IBM, which has a large network of support professionals.
• Compatibility: AIX is compatible with a wide range of hardware and software.

“However, AIX is also more expensive than Linux. It is also not as widely available as Linux. Ultimately, the best operating system for you will depend on your specific needs and requirements. If you need a stable, reliable and secure operating system for demanding workloads, then AIX is a good option. If you are looking for a more affordable and widely available operating system, then Linux is a good option.”

A lot of that sounds reasonable. But the part about AIX being compatible with a wide range of hardware and software? Not so much.

OSes on IBM Power

I persisted. I asked what operating systems run on IBM Power. Bard answered:

“The IBM Power servers can run a variety of operating systems, including:

• AIX: AIX is a Unix-based operating system developed by IBM. It is known for its stability, reliability and performance.
• Linux: Linux is a Unix-like operating system that is open source and free to use. It is a popular choice for servers and other high-performance computing applications.
• IBM i: IBM i is a proprietary operating system that is designed for business applications. It is known for its stability and security.
• z/OS: z/OS is a mainframe operating system that is developed and marketed by IBM. It is known for its scalability and reliability.

“Other operating systems: There are a number of other operating systems that can be run on IBM Power servers, including Solaris, HP-UX, and FreeBSD.”

Again, this was fine until the mentions of z/OS, Solaris, etc.

Key Takeaways

By this point, I’d seen enough. For anyone who’s worked with Power Systems and AIX, the errors I encountered are easy to recognize, but of course there are plenty of instances where I actually need answers. I guess the lesson is don’t accept anything at face value. Sure, I probably could have better phrased my questions. And I expect that AI tools will, in the near future, rapidly progress in their efficiency. But whether we’re querying a traditional search engine or an AI system, we still need to fact check our results.

While many, many AI think pieces have been posted just in the past few months, I like the perspective offered by the author John Scalzi. He recently conducted his own experiment with Bard.

“I asked Bard, Google’s AI, the name my upcoming novel, and also (as a test, because I haven’t publicly announced the title), the title of the novel after that. Bard correctly noted the name of the next novel (Starter Villain) and correctly noted I hadn’t publicly named the novel after that. But then, unprompted, it confabulated the following:

“Here are some details about Starter Villain:

• The book is set in a world where villains are a legitimate career path.
• The main character, Jason, is a young man who dreams of becoming a villain.
• Jason enrolls in a villainy academy, where he learns the skills he needs to be a successful villain.
• Jason’s classmates include a group of misfits and outcasts, who he quickly befriends.
• Jason and his friends must face off against a powerful villain, who is determined to destroy them.”

“None of the above is even remotely true. I mean, it sounds like a fun YA book! But that’s not the book I wrote.

“Why does Bard think this is what my book’s about? Got me. It’s not like the details of Starter Villain aren’t out there at this point; reviews are starting to come in and we have a “back cover” book synopsis on every online retailer out there.”

Scalzi concludes: “Today, I think there will be a whole generation of people, particularly my age and older, so used to the idea that Google and other search engines pull up ‘correct’ information—an idea promoted by Google and other search engine owners, to be sure—that they won’t even question whether the information they’re being offered up has any relation to the truth.

“AI will make the internet even less truthful than it is today. It is already doing it.”

Lest you think that Scalzi was being hyperbolic at the end, a law firm was recently found to have conducted research using ChatGPT that referenced non-existent example legal cases.

Maintaining a healthy skepticism was a good idea at the dawn of the public internet. If anything, it’s even more important now.

Nigel Griffiths: An AIX Content Machine

Here’s the latest from Nigel’s AIXpert blog. You’ll also find a link to his ever-expanding trove of YouTube videos.

Originally published by TechChannel May 17, 2023

Rob McNelly with the latest on invscout, along with tips, resources and an opportunity to give feedback to IBM

I’ve mentioned Inventory Scout (invscout) quite a bit over the years. That includes producing this video tutorial and citing this script.

So why am I bringing it up yet again? It’s to let you know that IBM has once again changed the location where you can download the catalog.mic file. Because IBM no longer uses FTP, I had to run this command to download the file:
            wget –no-check-certificate

Here’s how to upload invscout microcode survey (mup) files.

Survey files from Inventory Scout may be concatenated together and uploaded in one HTTP POST operation. Extra blank lines between survey files are ignored, however if a newline character is missing from the end of one of the survey files, the resulting concatenation may be unparsable. So it is a good idea to insert an extra blank line between files. The name of the upload file must either be mdsData, or end with .mup .

The prepared upload file may be sent to the MDS upload server by invoking a utility that performs a standard HTTP POST operation. A number of utilities are available on the internet that can perform the POST operation. For instance the following cURL command (available in the AIX Toolbox) will upload a file called “local.mup” to MDS for analysis:
            curl -F “mdsData=@local.mup;type=multipart/form“

If the POST goes correctly, the resulting output stream will contain an HTML document containing an analysis of the uploaded microcode survey file.

That may seem like a bother, but invscout is a very quick and effective way to determine if your server and I/O firmware are up to date. If I use my script and give it a list of hosts to check, I can get a consolidated report that provides tons of information: hostname, IP address, server model, serial number, and installed microcode levels (along with the latest available levels). Rest assured, invscout is worth the effort.

Network Adapter Issue Explained

IBM’s Darshan Patel recently posted this explanation of a PCIe3 connection issue.

Question: Why does the link of ent4 and ent7 in following setup take a long time to come up or not come up at all?

Discussion with Cisco revealed that the port connected to the Mellanox adapter needs special tuning. Mellanox adapters and switch use a low frequency communication method for auto-negotiation during the link up process. Some switches have compatibility issues and do not support the low frequency communication in their hardware. In order to overcome the switch port speed getting locked to the negotiation signal, Cisco Nexus 9000 switches have a dfe-tuning-delay command that enables them to start locking to the signal only after a predefined delay time to avoid trying to lock on the low frequency signal. There are signal paths placed on the switch PCB that connect switch port to the chip inside the switch. Not all the paths are the same length. Depending upon the signal path, only certain ports on the switch require this tuning.

Be sure to read the whole thing. And thanks to IBM’s Chris Gibson for bringing this to my attention.

Power Systems Prereqs

IBM has a web page called Power Systems Prerequisites.

By selecting your machine type, feature codes, and operating system, you can receive some valuable information. For example, I entered 9009-22G, Feature Code 5729, and ALL operating systems, and got back 18 prerequisites. Here are a few other returns I got, all with 5729 as the feature code:

            AIX Version 7.1 with the 7100-05 Technology Level and Service Pack 7100-05-06-2028
            (#0000) For any I/O configurations

            AIX Version 7.2 with the 7200-03 Technology Level and Service Pack 7200-03-06-2038
            (#0000) For any I/O configurations planned availability February 19, 2021

            AIX Version 7.2 with the 7200-04 Technology Level and Service Pack 7200-04-02-2028
            (#0000) For any I/O configurations

            AIX Version 7.2 with the 7200-05 Technology Level
            (#0000) For any I/O configurations

            AIX Version 7.3 with the 7300-00 Technology Level
            (#0000) For any I/O configurations

            IBM i 7.1: RS710-10 OS & TR PTF Group (SF99707 Level 11)

            Fix Level: C7192710 + HIPER PTF Group (#0000) Requires latest PTFs and HW feature EB3U
            IBM i 7.1 Activation for S922

Power10 Systems Redbooks and More

These publications have been out awhile, but if you haven’t had an opportunity to get up to speed on Power10 systems, here are some places to start:

            * IBM Power E1050: Technical Overview and Introduction
            * IBM Power S1014, S1022s, S1022, and S1024: Technical Overview and Introduction
            * IBM Power E1080: Technical Overview and Introduction

While I’m at it, here’s an introduction to 7063-CR2 HMC configuration from IBM Support.

Power Research Program Seeking Feedback

IBM has some surveys they would like you to fill out: one for the Power Research Program, one for Power Server Energy Modes, and one regarding OS subscriptions.

Teaching the Next Generation

If you’re not familiar with the IBM Power Skills Academy (previously known as the IBM Academic Initiative), read about the ongoing efforts to provide education and training on Power Systems, AIX and IBM i at colleges and universities worldwide.

The IBM Power Skills Academy (PSA) has equipped educators at eligible colleges and universities throughout the world with the materials, technology and resources they need to teach their students IBM Power skills.

These resources include free access to IBM Power courses and our Power Academic Cloud.

PSA enables colleges and universities to enhance their computer science, information systems, engineering and business programs, resulting in students who can compete in the job market of any industry. All PSA materials and resources are available to student and instructor members at no charge!

The Power Academic Cloud provides remote access to Power systems running AIX, IBM i and/or Linux, for teaching and non-commercial research activities. Based on your unique requirements, the PSA team will build an environment on which you can teach the latest Power technology and business application strategies. The Power Academic Cloud is available, for no charge, to faculty and IT staff who are registered members of PSA.

Originally published by TechChannel April 27, 2023

Rob McNelly offers another anecdote about the value of testing changes

One advantage of being an IBM Champion is the communication we get from IBM. I’m subscribed to various newsletters that provide useful information. For instance, I was recently notified about a webinar about Ansible. This is from the email:

“Join us to learn how Ansible offers significant benefits by providing fast and repeatable installations and configurations for AIX and the Oracle Database, including the RAC feature. In this webinar, you will learn what the available AIX and Oracle Ansible automation tools are, and how they can be used to achieve better and more reliable installations.”

Here’s the replay.

That same newsletter includes links to the recently updated AIX strategy paper and roadmap, the community badge program, AIX training courses, and AIX community information. Finally, there is a primer on why “AIX running on IBM Power10 is a winning combination for your business.”

Sure, you may have already seen some of this information, but I appreciate these reminders.

On the subject of IBM resources, check out what’s new at IBM Support. Recent updates include details on opening a support case (in text and video formats), among many other topics.

CLI Options: An Update

A reader recently reached out about something I wrote for my AIXchange blog almost a decade ago. The post is about using the HMC command line interface (which I still love).

Apparently I was unclear with my examples, because the reader copied the first one, thinking it would add an additional adapter in his environment without affecting the adapters he already had. Instead, it removed his existing virtual adapters. Fortunately, he was able to save his running config and recapture the virtual adapters that had been removed.

Remember: When you see something online, no matter the source, make sure you try it out in a test/dev environment first before putting it into production.

I should add that I’ve since found a simpler way to add virtual adapters and VLANs. Check out this TechChannel column from 2021.

Connecting eBMC Power Servers to the HMC

If you’re looking to connect your Power10 eBMC system to an HMC, this document may help:

“eBMC Power servers introduce a new connection type between HMC and managed server. Each eBMC server requires two connections; one to the BMC itself and one to the Virtual Management Interface (VMI). This requires two IP addresses per BMC Ethernet port. As with FSP-based systems there are two eBMC Ethernet ports to allow for redundant HMCs. Each HMC is cabled (private DHCP networks) or connected via static IP address to one eBMC port. Redundant HMCs require a total of 4 IP addresses for each server. The default setting for the eBMC IP address is DHCP client, however the default setting for the VMI IP address is currently set to static. The VMI IP settings should be configured before powering on, or the server will go into a no connection state when it reaches standby. The eBMC IP address is active when the server has completed AC power apply initialization (power off state). The VMI IP address is not active until the server is powered on (standby or operating).”

HMC Enhanced UI Configuration

Another HMC item, this one on the Enhanced UI. It’s from 2020, but the information is relevant:

“With Enhanced UI, we simplified the virtual network management experience and designed a simple and fresh model that helps you configure networking to the VMs with few simple steps avoiding the chances of user errors (and all that from a single console). With the newer model, configuring network adapter for a VM is as simple as creating a network on the system and adding VM to that network (that’s it!)”

Call Home Users Need to Update Their Connections

This notification has made the rounds, but in case you haven’t seen it, there are significant changes are on the horizon with IBM’s Call Home electronic fix distribution solution:

“Public internet IP addresses are changing for the IBM servers that support Call Home and electronic download of fixes for customer system’s software, hardware, and operating system. This change pertains to all operating systems and applications connecting to IBM for electronic Call Home and fix download. Customer action might be required to ensure uninterrupted Call Home and fix delivery services.

“New network connections between your machine and IBM servers are required to keep your ability to perform Call Home and download fixes. If you have a firewall in your network, you might need to make changes to allow the new connections.”

According to the table in the IBM Support link, most of these redirects will occur in 2024. But depending on your change control and how siloed your organization may be, it’s advisable to take action at your earliest opportunity.

Originally published by TechChannel March 23, 2023

Rob McNelly explains how he recently updated an older HMC with the help of IBM Support and gives a reminder about technical debt

In January IBM announced its latest class of IBM Champions.

As is noted in that blog entry, “This year’s 839 IBM Champions come from 60 countries; 68% of them have been IBM Champions before and are returning to the program, including our 30 Lifetime IBM Champions.”

That I am one of those 30 Lifetime Champions is still humbling for me, but of course it’s an honor to be associated with all the Champions. And there is an ongoing association, via the IBM Champion Slack channels. I get on whenever I can. Being able to interact in near real time, to ask questions of and get advice from these very smart people located around the world, is pretty cool.

IBM Support to the Rescue

I was recently asked about upgrading an older HMC. Writing for this website, Jaqui Lynch has a lot of good general information about upgrading and maintaining your systems.

But once I found something relevant to the specific question, courtesy of IBM Support, I thought I had a handle on it. That’s because when I read this:

“This document provides instructions to upgrade vHMC for PowerVM and HMC 7063 machine type from Version 9 Release 1 with mandatory fix MH01858, or V9R2M950 to V10R1M1010.”

I felt confident that I could apply MH01858 and then go directly to V10R1M1010 without messing with any V9 versions.

Simply opening a ticket with IBM Support saved quite a bit of trouble, as they were able to confirm that my client could indeed make the leap. Unfortunately, a hardware issue unrelated to the upgrade threw us for a bit of a loop.

We proceeded through the steps. But once we ran:

            chhmc -c altdiskboot -s enable –mode upgrade

and then:

            hmcshutdown -t now -r

nothing happened.

Support had us go through the process again. When that didn’t work, they asked us to send a debug file.

At this point, they determined that somehow the RAID1 array kicked out one of the drives and sent us a procedure to resolve the issue. This involved getting the root password from Support and running various commands, including:

            /opt/hmc/bin/mvcli info -o vd
            /opt/hmc/bin/mvcli delete -o vd
            /opt/hmc/bin/mvcl rebuild

The point is, if you run into this type of issue, you will most likely need the help of IBM Support, so I’ll leave you to open your own ticket and get the procedure.

Once the array was rebuilt and Support confirmed there were no errors, we were able to continue with the firmware upgrade, the upgrade to V10R2M1030 and the managed system firmware.

Around that time, I pressed a button on my iPhone to upgrade the software. That process went just a bit more smoothly.

A Script to Collect HMC Profile Data

Also HMC-related: Kristian’s Blog: HMC Profile Diff

“More often than not, I find myself having to compare two Hardware Management Console (HMC) logical partition (LPAR) profile configurations. Sometimes this is to ensure that a profile in a disaster recovery site matches that of its production counterpart. Other times, it’s to make sure all members of a cluster have identical resource configurations.

“In a perfect world, I’d be managing the LPAR configurations using something like Terraform, but this currently isn’t an option. I can login to the HMC and manually verify the profiles (which is what I’ve been doing), but this is tedious and prone to error when you need to compare many profile pairs.”

A Reminder About Technical Debt

One of the interesting things about consumer software is that you have few choices when it comes to upgrades. While you can postpone updates on Windows machines, or your phone, the fixes will load eventually. Where am I going with this? The same place I always do. Please consider upgrading your machines on a regular schedule to avoid technical debt.

SXSW is Going On, But There’s an Even Cooler Scene in Austin

Many of us who work on AIX have been fortunate enough to have spent time at the IBM facility in Austin, Texas. But if you haven’t been on site, see it for yourself, and learn how the E1080 models were developed and tested.

Originally published by TechChannel February 16, 2023

Rob McNelly covers how to troubleshoot common NIM issues and AIX tips and tricks

I enjoyed this perspective on the Cisco Discovery Protocol (CDP) from IBM Champion Andrey Klyachkin:

“You know this eternal problem, don’t you? Datacenter guys laid the cable from an AIX server to some port on some network switch. Of course, everything was written on some sheet of paper many years ago. But years went after years, the paper was blown away by the wind and nobody knows anymore where the cable goes to.

“Fortunately Cisco sends CDP packets regularly and you can find the information about the connected port and switch there. I even know datacenters where it is still allowed and some where security guys asked to shut down CDP…

“If CDP is switched on, it sends a packet every 60 seconds with the information about the switch and port. You can get the packet and decode it to get the information. The only problem is to decode the information.

“One of the new features of IBM AIX 7.2 TL5 is CDP daemon. AIX understands and can decode CDP packets. In order to use it, you must have the packet bos.net.tcp.cdp installed…”

Read the entire post, which includes four screenshots, on Andrey’s LinkedIn feed. There’s also a caveat about running cdpctl show port enX to get the information you’re interested in. All in all, this seems much easier than the method I wrote about back in 2014.

While that was posted back in December, the conversation continues. Read this response via Twitter, along with this article on implementing the Link Layer Discovery Protocol (LLDP) on VIOS.

Finally, here’s a breakdown of the differences between LLDP and CDP:

“LLDP is a layer two discovery protocol, similar to Cisco’s CDP. The big difference between the two is that LLDP is a standard while CDP is a Cisco proprietary protocol.

“Cisco devices support the IEEE 802.1ab version of LLDP. This allows non-Cisco devices to advertise information about themselves to our network devices.”

Diagnosing Hanging NIM Operations

As a long-time NIM advocate, Chris Gibson’s tweet on troubleshooting hanging NIM mksysb caught my eye. You’ll find the details in this IBM Support document:

“Problem: In this document we go through a few of the most common causes for a hanging mksysb operation. We will dive into the internals during NIM backups and learn how to troubleshoot and fix those problems.

“Symptom: The mksysb displays 100% complete on your NIM, but prompt is never returned. The mksysb is hanging at a certain percent of the backup.

“Cause
1 The cause for a hang is usually a problem with the network. At the beginning of a NIM mksysb, the NIMSH daemon working on the client LPAR will open two TCP sessions, one on client port 3901 to > NIM 1023-513 and one on client port 3902 to > NIM 1023-513 where the second session is referred to as Auxiliary session and will be used to relay the mksysb command success/failure return code when the backup complete. If this session is dropped or interrupted, the NIM master will keep waiting for that return code even after the process is fully complete and successful.

2 During mksysb backup we use the ‘backbyname’ command to back up the data we need, if the command is unable to access/read a specific file or directory, the process may hang. Normally, this would be caused by a hung NFS mount point or one where the root used has no read permissions for. Additionally, this may be caused by a corrupt file system.”

AIX USB Device Quirks

Here’s yet another tip that came my way via Chris on Twitter. While one could make a strong case that AIX pros are quirky, this information from IBM’s AIX 7.3 document archive covers quirks:

“Starting from AIX 7.3 Technology Level 1, the AIX operating system provides quirks to support various third-party USB mass storage devices.

“The following quirks are supported starting from AIX 7.3 for USB mass storage devices:

delay_doorbell: If this quirk is set, the ringing of the adapter doorbell is delayed by 1 ms.

cbw_csw_order: If this quirk is set, the data packets are sent only in the order of command block wrapper (CBW), data, and command status wrapper (CSW). Most of the third-party devices, for example, Seagate and Western Digital, need this quirk to operate correctly with the AIX USB stack.”

Also explained: how to add quirk for USB devices, and how to create quirk entries for specific USB devices, be they from third parties or any specific vendor.

Originally published by TechChannel January 20, 2023

Rob McNelly shares clients’ stories and details how he solved their technical challenges

Recently, I helped mount a filesystem on a Windows system that was being exported from an AIX system. The client wasn’t sure it would work—and there were challenges—but fortunately I found some guidance by googling.

Following these instructions, I was able to export a test filesystem from an AIX machine in my lab. Then I NFS mounted it from a test Windows machine.

Since I wasn’t running a Windows Server OS version, I ran the command listed in the document under Desktop OS in a PowerShell window:

Enable-WindowsOptionalFeature -FeatureName ServicesForNFS-ClientOnly, ClientForNFS-Infrastructure -Online -NoRestart

I couldn’t run the mount command in the PowerShell window, but I was able to run it in a normal command prompt window:

mount -o anon 10.1.1.1:/robtestfs z:

Initially, running the mount command triggered Network Error = 53. To fix that, I added the Windows machine as a temporary entry on the AIX machine in /etc/hosts. This allowed AIX to resolve the Windows machine hostname and mount the filesystem.

Another Reason to Keep Your System Firmware Current

A client wanted to update their VIO servers. After running shutdown -restart, the VIO LPAR would hang with LED code CA000040. This time the internet led me to a familiar source: IBM Support, which gave me these instructions:

“Problem: LPAR boot on a POWER9 system may hang with code CA000040. In iqyylog SRC B200F003 might be reported.
Symptom: LPAR does not boot and no access to SMS is possible.
Cause: This issue is caused by a new timer variable implemented at POWER9 used for creating delay timers. All POWER9 systems would be exposed, if those servers are not restarted for 814 days.
Environment: Any POWER9 system. The solution is provided in FW930.30, FW950.00, and FW940.30.
Diagnosing The Problem: To verify how long a system has been running, check the SRC history via ASMI with celogin:

“System Information -> Progress Indicator History

“Check if the system is up >= 814 days based on the time stamp on the STANDBY entry.
“A nondisruptive resource dump would include this information also.

“Resolving The Problem: Workaround is to set the LPAR processor compatibility mode to Power8 until firmware fix is installed. This needs to be done for any LPAR facing the problem.

Powering the system off/on will reset the timer.”

Me again: In this case, the client changed the VIO profile so that it ran in POWER8 processor mode. This allowed the LPAR to boot. Then they later scheduled an outage to update the system firmware, so this shouldn’t be an issue going forward. If you have POWER9 systems that have been up for a long time sans firmware updates, keep this possibility in mind.

Seriously, this experience makes the case for regular patching cycles. Even if the client had only patched once a year, this issue could have been avoided. Don’t forget: You don’t just patch the OS. Also consider system and device firmware, the HMC, etc. There’s nothing worse than worse than opening a ticket with IBM and being told that a months-old fix pack or a firmware update would have solved your problem.

Using NIM to Install VIOS

While the HMC offers its own VIO server installation options, for many of us, NIM is still the way to go. IBM Support provides additional information about NIM for VIOS installation:

“This document guides you through the steps of preparing NIM for new VIOS installation. It provides instructions how to create the needed NIM resources to perform VIOS installation. In case you run into any problems during this operation open new case with IBM Support team for further investigation.

“This topic is covered in the following parts:
Downloading the VIOS iso image
Pulling the mksysb_image file from the VIOS iso image and checking its integrity
Defining the VIOS as client of NIM
Defining mksysb and spot resources and allocating them to the VIOS
Booting the VIOS in SMS over network”

This doc also features several helpful screenshots.

PowerVM and AIX Networking Options

This IBM Support document lists some quick tips for configuring PowerVM and AIX:

“The purpose of this document is to list common issues and their solutions concerning Ethernet adapters with IBM PowerVM and AIX. This document covers dedicated (stand-alone) Ethernet adapters, EtherChannel and SEA configurations.

“This document discusses the following topics:
Link Down and Link Up errors on Ethernet devices
General EtherChannel Failure
Link Aggregation Control Protocol (LACP) EtherChannel Tips
General Shared Ethernet adapter (SEA) failures
Configure output options for unused Ethernet devices”

Power10 to Utilize Oracle Databases?

An interesting development:

“IBM has quietly announced it is planning a 24-core Power10 processor, seemingly to make one of its servers capable of running Oracle’s database in a cost-effective fashion.

“A hardware announcement dated December 13 revealed the chip in the following ‘statement of general direction’ about Big Blue’s Power S1014 technology-based server:

“IBM intends to announce a high-density 24-core processor for the IBM Power S1014 system (MTM 9105-41B) to address application environments utilizing an Oracle Database with the Standard Edition 2 (SE2) licensing model. It intends to combine a robust compute throughput with the superior reliability and availability features of the IBM Power platform while complying with Oracle Database SE2 licensing guidelines.”

A Closer Look at the Latest AIX Release

If you didn’t catch it ahead of the holidays, Chris Gibson wrote something for this very website. Be sure to read his thoughts on some AIX 7.3 TL1 enhancements that caught his eye.

More Thoughts on the Cloud Journey

So last month, I drove cross-country. More recently, my wife and I were traveling again, this time by air.

It was fascinating to look at the map as our plane covered the miles in literally fractions of the time that it did by car. Going from Amarillo to Albuquerque took many hours by car, but in flight that portion of the trip felt like it was only a matter of minutes. Sure, that’s to be expected when you’re moving at 500 MPH, but it’s still impressive.

Don’t forget boats: Years ago, I took an automobile ferry across Lake Michigan. That was unique, and of course it was also a time-saver compared to the alternative of driving around that Great Lake. I’ve yet to make long journeys by train. Of course, passenger rail travel is more common in Europe and elsewhere than here in the U.S.

So which method of travel is best? Of course, that depends. As much as I hike and camp, I can’t see me piling my belongings into an RV and seeing the country for months at a time, but plenty of people do just that. But how you choose to get there comes down to your preferences and circumstances. What kind of cargo are you hauling? How much time do you have? How much do you want to see new scenery and experience new places? How much do you hate lines at airport security checkpoints?

Last month I compared traveling options to businesses exploring their options with cloud. I still like the analogy. Should you keep your systems on-premises? Should you move only certain workloads? Should you contract with a managed services provider? It depends. These are questions only you and your organization can answer. You may find others’ experiences with cloud instructive, but ultimately, no one knows your business like you. It’s up to you to determine the best way to your destination.

Originally published by TechChannel December 6, 2022

Rob McNelly on cloud technology, expanded IBM Support options, IBM requiring encryption for fix downloads and AIX history

Recently my wife and I traveled cross-country via car. We spent time with family in Virginia and then drove north through Maryland, Pennsylvania and New York state. After celebrating the Thanksgiving holiday, we circled back through Ohio, Indiana, Illinois, Missouri, Oklahoma, Texas and New Mexico before returning home to Arizona.

Over two weeks, we covered thousands of miles, encountering varying weather conditions—fog, rain, snow—and stages of road construction. I guess spending that much time on the interstates made me a bit loopy, but I maintain that “Uranus Fudge Factory” is a funny name, and their billboards made me legitimately curious about the quality of their fudge.

I used Google Maps to navigate the journey. Initially we avoided the tolls, which took us to some interesting back roads and out of the way places. But as we got closer to home, we opted for the most direct route, never mind the cost. Toward the end I know I was just ready to be done with the drive.

Still, I’m glad we didn’t fly. For the most part I enjoyed the journey and the changing scenery. Driving naturally gave me time away from my phone, my email and work in general. It gave me a chance to think—though for better or worse, that eventually led me back to work considerations.

One thing I kept thinking of was companies’ adoption of cloud technology. It’s like driving or flying, at least in the sense that there’s more than one way to get there. It could be a quick trip, like a plane flight, or a longer road. 

Some companies want everything to remain on-premise. An onsite data center and IT staff is what they know; it’s what’s worked. In a sense, their journey may never begin. Other companies are taking action by enabling rapid provisioning or live partition mobility. Theirs might only be a short journey, but they’re moving forward.

Then there are true hybrid cloud environments, where software and/or infrastructure may be deployed as a service. Other companies no longer manage their own servers. Everything has gone to the cloud. Or they may still own the servers but pay service providers to manage them day to day.

In short, we all have different needs and priorities. Every company has unique goals. So of course every cloud journey is different.

A Change From IBM Support 

Just as there are pros and cons to driving versus flying, there are pros and cons to the many available options for supporting your business. One size may not fit all, but it is good to have options.

On that note, IBM Support recently announced that it is providing customers with greater flexibility to add and invite team members to work on support cases:

“Coming soon, users on your accounts will have new flexibility over who they can invite as team members on support cases. New team members do not need to be associated with the account or have an IBM ID.

As an administrator, you will no longer be required to take the time to approve new team members. While this change gives users more freedom to add team members, you will still receive email notifications when someone on your accounts adds a team member. You can still use the User Administration page to remove users you don’t agree to from your cases, just as before.

Here’s how it works
When a user creates a case, they simply select Add A Team Member from the Add Team Members menu, then enter the new team member’s name and email address, and then click Search.

If the person they enter does not already have an IBM ID, the person opening the case clicks OK to confirm that they’d like to invite the new team member to the case.

The people added to cases with this method will have full access to the case they are added to. They will be able to add comments and edit the case. They will not have access to any other cases.”

Support Ending for Unencrypted Fix Downloads

IBM recently announced that it will no longer support unencrypted fix downloads. This change is set to go into effect on February 15, 2023.

From IBM Support:

“Many leaders of the internet industry—such as World Wide Web Consortium (W3C), Internet Engineering Task Force (IETF) and Internet Architecture Board (IAB)—state that universal use of encryption is the way forward for the internet traffic.

Therefore web platforms should be designed to actively prefer secure communication so data is protected in transit and at rest.

Aligned with this industry direction, IBM IT Security Standards have been enforcing the use of encrypted communications.

Therefore IBM Electronic Fix Distribution (EFD)/IBM Electronic Customer Care (ECC)/IBM Fix Central systems will stop supporting unencrypted fix downloads on February 15, 2023, to improve user privacy and security and enforce compliance with IBM IT Security Standards. Shortly after that date, unencrypted fix download flows will NOT be allowed anymore.

Recommended Action
Ensure as soon as possible that the connections made to IBM fix download servers are secured.

The IBM fix downloads servers currently support HTTPS, SFTP, FTPS and DDPS secure download protocols. Ensure you have secure protocols in place and update any procedures, including existing jobs, scripts or tools to use the secure fix download protocols.

You can use secure protocols now, but they will be the only options when they are enforced on the deadline specified above.

If you are still using an unencrypted fix download protocol (such as HTTP, plain FTP or DDP) then make sure you switch to an encrypted one (such as HTTPS, SFTP, FTPS or DDPS).”

A Brief History of AIX 

Nigel Griffiths recently updated this document that features release/end of support dates and notes for each AIX version from 3.1 through 7.3. The doc also includes a brief AIX timeline as well as tips and other “often forgotten” details.

Originally published by TechChannel October 19, 2022

Have you heard of “lights-out data centers?” Rob McNelly explains what they are along with their pitfalls, and explores the latest IBM announcements here.

Though I’ve heard about lights-out data centers for years, I truly don’t envision a future where humans will never set foot on the raised floor. We’ll always need hands and eyes in the room to perform tasks on our systems.

Case in point: Recently I serviced a customer that had three of their four fibre network ports inactive on their network switch. For example, ent0 showed that we were disconnected:

                 entstat -d ent0
                  Link Status: Down
                  Media Speed Selected: Autonegotiation
                  Media Speed Running: Unknown

While ent1 was fine. We were connected:

                  entstat -d ent1
                  Link Status: Up
                  Media Speed Selected: Autonegotiation
                  Media Speed Running: 1000 Mbps Full Duplex

The OS was not seeing the expected connection on the network ports. This was verified by the network team, who could also see from the switch side that the ports they expected to have connections were in fact not connected.

During this call, we learned that this was an ongoing issue. The client initially tried replacing small form factor pluggables (SFPs) on the switch. They physically verified that the expected ports from the server were plugged into the correct ports on the switch.

We had the luxury of being able to swap cables, and lo and behold, the problem followed the cables. What was the working port prior to the swap ceased to function, and vice versa. Was it a bad cable? Nope. We tried a different cable and had the same issue.

At that point it was lights on, figuratively, in our heads, because we realized that the TX and RX polarity was reversed on the cable. So we asked the onsite team to correct the cable and plug it back into the switch. As expected, the port fired right up. All three of the non-working ports had this issue, so we did two more reversals of the TX and RX.

Working remotely, we could adjust the switch and logical configurations on the server all we wanted, but it wouldn’t have accomplished anything. To fix this problem, we needed people on site.

On that note, be sure to show your appreciation for the CEs and any data center personnel you work with. If you yourself are a “hands and eyes” person, then I thank you, too. Remember, without the professionals who work directly on these systems and associated equipment, none of us are doing much of anything.

First Impressions of IBM October 11 Announcements 

As IBM Champion Alan Fulton notes, there is indeed much to unpack with IBM’s October 10 announcements, starting with updates to PowerVM, vHMC, PowerVC, AIX and IBM i.

There’s plenty that caught my eye as well:

• Support for AIX install and boot from iSCSI attached storage. Consult the IBM System Storage Interoperation Center (SSIC) for additional information on supported configurations.
• Increased NFS file size limit beyond 32 TB. See the AIX 7.3 TL1 Release Notes for the new supported limits
• AIX tar command support for pax archive format. Previously, AIX tar supported only star archive format. The new pax format archive can be created using the “– format=pax” option in AIX tar command.
• Improvements in AIX dump performance through hardware-accelerated compression on IBM POWER9 and Power10 systems
• JFS2 filesystem now allows dynamic switching between inline and outline logging
• The chpv command now provides an option to force offline a poorly performing PV in a mirrored pair
• Ability to perform VIOS updates using vHMC

That’s just an abbreviated list. Read the announcement letter for yourself.

Two More Tales From the Field

Another customer had a VIO server that was spitting out unexpected vfchost errors in the error log, so they opened a ticket. IBM Support pointed them to this information: 

Problem: Qlogic or Cavium IBM fibre adapters in Power Systems register as targets in the SAN fabric instead of initiators.
Symptom: Any one of a number of symptoms might be present, including:

1. NPIV client LPARs do not discover devices during scans by system firmware (SMS or ioinfo)
2. AIX hosts can fail to boot
3. The AIX error log might be filled with extraneous errors when SAN monitoring software runs, or even when cfgmgr runs, as the adapter attempts to log in to itself. The errors decode as name server query failures. Detailed SENSE DATA indicates the failure was against the physical adapter’s own N_Port ID.

The doc also notes that, on a VIO server, these steps must be performed from the oem_setup_env prompt. Then reboot the VIO server. In our case, we followed the directions and the errors went away. So keep this in mind should you run across something similar.

And one final story: Yet another customer that uses SSH to access a server wanted to determine why the sessions would end when left open for some time. There are a few ways to deal with this problem, but start here, and scroll down for this response. 

ssh -o TCPKeepAlive=yes -o ServerAliveCountMax=20 -o ServerAliveInterval=15 my-user-name@my-server-domain-name-here

My customer tried that solution from their command line and it worked, and eventually they made the change to their /etc/ssh/sshd_config file so they no longer needed to enter those options on the command line.

IBM Support Forums Have Moved 

As of October 11, the IBM Support forums are now part of the IBM Community website:

“To improve your support experience and provide you with the best possible access to people who know and understand your products, the Support Forums join the IBM Community on October 11, 2022.

Simply visit the IBM Community website to search for and continue discussing your products there. The IBM Support site will provide a link to the IBM Community for some time after the move, but we recommend all users update bookmarks pointing to the Support site’s Forums as soon as possible. To make this transition as easy as possible, the Forums will remain on the Support site until November 11, but you will only be able to read questions and responses there, not post new ones.”

While I’m on the topic of IBM Support, be sure to check out the Complete Guide To Must Gather LPM Data Collection on PowerVC, VIO, AIX, Linux and IBM i.

The Latest From Nigel and Chris 

If you know AIX, you know Nigel Griffiths. And if you know Nigel, you know nmon is his baby. He recently sent out this information.

If you have thousands of nmon files, you can drown in the high volumes of data. You need to extract the key facts to allow planning your server consolidation, migrating to newer servers or Power Live Partition Mobility. These nsum shell scripts allow does the hard work to build a CSV file to import into a spreadsheet for further work.

Also via Twitter, Chris Gibson points to this document on migrating workloads to Power9 and Power10 systems. And on his personal blog, he explains how to find the hardware uptime for Power Systems frames.

“We needed to find the hardware uptime for a particular POWER9 frame to determine how close we were to hitting this known POWER9 firmware bug.

We found we could calculate this by looking at the “Progress Indicator History” view in ASMI and looking at the date associated with RUNTIME/STANDBY and working out how many days had passed since the frame was powered up.”

In that post he links to a C program that calculates the uptime, so check it out.

Originally published by TechChannel September 2, 2022

Rob McNelly highlights tips on open firmware macros, recaps Nigel Griffith’s closer look at Power10 Servers, explores AIX security bulletins and more

There are several ways to boot LPARs on IBM Power Systems servers. For example, you can boot to your OS of choice—AIX, IBM i, or Linux—or to SMS or open firmware. While most of us are familiar with all of these options, if you’re not, check out this IBM Support document. I’ll highlight the section on booting to the open firmware prompt:

Booting the LPAR to the Open Firmware (OK) prompt
1. Make sure the LPAR is not activated. If it is hung, go to the HMC GUI, and under Systems Management -> Servers -> server name, check the box next to the LPAR. Then, from the arrow on the right side of the LPAR name, activate the menu and select “Operations -> shutdown”.
2. Wait until the LPAR is in a “Not Activated” state, and the Reference Code shows all zeros.
3. Mouse click the arrows to the right of the LPAR name again to display the menu. Click “Operations -> Activate -> Profile”
4. From the Activate Logical Partition window, click the “Advanced” button.
5. From the Activate Logical Partition – Advanced button window, select “Open Firmware OK Prompt” from the “Boot Mode” drop down list.

Chris Engel has information about the firmware chain of trust for PowerVM, while Colleen Stoufer explains the restricted OF prompt.

Access privileges at the Open Firmware prompt must be restricted to preserve the secure boot status of the PFW code loaded on the partition. The new Restricted OF Prompt will limit input and execution to a defined set of macros. The option to access the Restricted OF Prompt will be displayed on the splash screen. You will see that the “8 = Open Firmware Prompt” option has been replaced with the “9 = Restricted Open Firmware Prompt” option.

Once you’ve booted to the restricted OF prompt, you can do many things, as you’ll discover by consulting the Restricted OF Prompt User Guide. (Hat tip: Chris Gibson on Twitter.) This is one handy doc, because it allows you to learn more about your system before you’ve even loaded an actual OS:

“When a partition is booted in Firmware Secure Boot mode, normal access to the Open Firmware prompt is disabled. A new Restricted OF prompt is provided that allows access to a set of macros that will allow customers to continue to perform many of the functions that they rely upon without jeopardizing the security of the firmware. This prompt will not allow execution of any commands that are not part of this documented set of macros. Existing tools will require updates to function in this new environment.”

The set of macros defined in this section will execute at the Restricted OF prompt. The inputs and expected output are described. In most cases, the macros will output a string which indicates the success or failure of the execution of the macro.

** The set of macros is subject to change. New macros will be added as required, and defunct macros will be removed. This document will be updated when changes are made.

In the following sections that describe the macros, the following will apply:
• All input at the Restricted OF prompt is case insensitive.
• Square brackets are used to enclose any optional items.
• All parameters for the macros MUST be on the same input line as the macro name.
• The default language for the macros is English (no translations).

Try running macro_help. This macro displays the list of the currently supported macros and the required parameters.

0 > macro_help
>> BOOT_FROM_SEQ [ADDPARMS]
 where:
 [ADDPARMS] = additional parameters such as debug flags
 (if not provided, boot from devices in boot-device list without debug)
>> DISPLAY_BOOTSEQ
>> SET_DEFAULT_BOOTSEQ
>> BOOT_FROM_DEVICE <DEVTYPE> <ADDRESS> [BOOTPARMS]
 where:
 DEVTYPE = { #disk | #cd/dvd | #san | #network | #tape }
 ADDRESS = location-code
 [BOOTPARMS] = specific to the device type
 (see Restricted OF Prompt User Guide for examples)
>> DISPLAY_BOOT_DEVICES <DEVTYPE>
 where:
 DEVTYPE = { #disk | #cd/dvd | #san | #network | #tape | #all }
>> DISPLAY_MAC_ADDRESS <ADDRESS>
 where:
 ADDRESS = location-code
>> DISPLAY_NETWORK_PATHNAME <ADDRESS>
 where:
 ADDRESS = location-code
>> PING <ADDRESS> [PINGPARMS]
 where:
 ADDRESS = location-code
 [PINGPARMS] = required and optional parameters
 (see Restricted OF Prompt User Guide for examples)
>> DISPLAY_PCI_PROPS [ADDRESS]
 where:
 [ADDRESS] = location-code
 (if not provided, all adapter PCI properties will be displayed)
>> DISPLAY_ADAPTER_WWPN [ADDRESS]
 where:
 [ADDRESS] = location-code
 (if not provided, all adapter WWPNs will be displayed)
>> LUN_ATTACHED? <ADDRESS>
 where:
 ADDRESS = location-code

Usage examples are also included, so be sure to check it out. Download the PDF. 

More Power10 Deep Dives

During a recent Power Systems Virtual User Group webcast, Nigel Griffiths covered the Power10 scale-out and midrange systems. Listen to the replay and download the presentation. 

Nigel has another webcast coming up this week. He takes a closer look at the Power10 S1024 scale-out system. In addition to displaying tons of photos, Nigel will discuss the new service processor and eBMC. He’ll also touch on the mandatory HMC v10, which is used on Power10 hardware, including the new S1022/S1024 and E1050 and the previously released E1080.

Best Practices Doc Updated

The latest version of the Power Implementation Quality Standard document is now available. I’ve mentioned Fredrik Lundholm’s work before, most recently here. Most slides have a headline in red noting the most recent update. This being Version 2.6, the most recent updates are labeled “Upd 2.6.” And be sure to check out the notes at the bottom of many of the slides; you’ll find good information there as well.

Recent AIX Security Bulletins

I assume you are keeping up with AIX security bulletins. Three recently caught my eye:

• AIX is vulnerable to arbitrary command execution (CVE-2022-1292 and CVE-2022-2068) or an attacker may obtain sensitive information (CVE-2022-2097) due to OpenSSL
• IBM PowerVM VIOS could allow a remote attacker to tamper with system configuration or cause a denial of service (CVE-2022-35643)
• AIX is affected by multiple vulnerabilities in Python

Be sure to keep your systems patched, and sign up for these security bulletins if you are not already receiving them.

Originally published by TechChannel August 2, 2022

The availability of the full Power10 server portfolio is another reminder of the noticeable benefits of moving forward with technology

Recently, after settling into a new home, I upgraded my cable modem and router. I was fine with the internet service I had, but for the same cost, I was able to go from ~100Mbps to ~1000Mbps download speeds. Obviously those numbers may vary depending on your connection (wired versus wireless), your equipment, and the interference from other wireless devices in your area, but rest assured, upgrading is worth it. Browsing is faster. Moving large files around is faster. The overall experience is better, and the difference is very noticeable.

The same thing happened earlier this year when I upgraded my phone. I had no complaints about the old phone, but the improved performance was strikingly apparent once I made the change.

That’s the basic story of technology and its impact. The devices we use day to day serve us very well, and may do so for years on end, but once you finally make the move, you’re immediately reminded how quickly technology advances. As I’ve noted previously, it really sneaks up on you: 

“So maybe you need a new laptop. Or maybe it’s time to look at your infrastructure and consider upgrading your hardware and software. Again, I was fine chugging along as I was, but now with the snappier performance, I realize what I was missing out on. I also can’t count the number of times I’ve seen IBM Power Systems users react similarly to the performance of new hardware. Yes, everyone has budgets, but what is it worth to your organization when response times are better, jobs complete faster and more work is being done with fewer cores?”

Upgrading to Power10

With that, let’s talk Power10. The entire server portfolio, both enterprise and scale-out, is now available. As you would expect, the performance has significantly improved yet again. Do you think you’d notice the difference if your servers were upgraded?

Here’s the IBM press release. In addition, I received this IBM email that provides basic details:

Today we are announcing the rest of the Power10 server family; the scale-out Power S1014, Power S1022, Power S1024, and the midrange Power E1050. These new systems, built around the Power10 processor, have twice the cores and memory bandwidth of the previous generation to bring high-end advantages to the entire Power10 product line.

IBM Power S1014

The IBM Power S1014 is a 1-socket, 4U Power10-based server for IBM AIX, IBM i, and Linux workloads, and has 57% more performance per core and 20% more memory bandwidth compared to the Power S914. Reduce physical data center footprints and lower your cooling and electrical costs by doing more with less. The Power S1014 is ideal for IBM i, Oracle Database SE, AI inferencing, and more. Learn more about the Power S1014 and tour a virtual demo.
 
IBM Power S1022

The IBM Power S1022 is a 2-socket, 2U server for IBM AIX, IBM i, and Linux workloads, and has 37% more performance per core and 2.4X more memory bandwidth compared to the Power S922. It is available in either a single chip model, the Power S1022s, or dual chip model, the Power S1022. The Power S1022 is ideal for distributed computing, DevOps, dev/test environments, and more. Learn more about the Power S1022 and tour a virtual demo.

IBM Power S1024

The IBM Power S1024 is a 2-socket, 4U server for IBM AIX, IBM i, and Linux workloads and has 33% more performance per core and 2.4X more memory bandwidth compared to the Power S924. With double the number of cores compared to Power9-based servers, you can lower your cooling and electrical costs by consolidating more workloads onto fewer servers. Users can also further optimize and reduce costs by taking advantage of flexible consumption models and only paying for what they use. Learn more about the Power S1024 and tour a virtual demo.

IBM Power E1050

The IBM Power E1050 is a 4-socket rack server optimized for data-intensive applications and hybrid cloud deployments. Enhanced security with transparent memory encryption and production-ready AI at the point of data enable faster insights for clients. Scaling is consistent across private and public cloud environments with flexible consumption options for users. Learn more about the Power E1050 and tour a virtual demo here. 

If you want to go further in-depth, here are some things I’ve come across, starting with this blog post by Ken King, General Manager, IBM Power:

“We introduced the IBM Power10 high-end server last September and we are continuing to broaden the portfolio with a major launch of four new systems today: the scale-out Power S1014, Power S1022 and Power S1024, along with a midrange server, the Power E1050. These new systems, built around the Power10 processor, have twice the cores and memory bandwidth of the previous generation to bring high-end advantages to the entire Power10 product line.”

For other perspectives on the new servers, start with this very website, which has an announcement feature and Lifetime Champion Jaqui Lynch’s analysis.

Nigel Griffiths’s summary includes links to IBM Redbooks, rperf numbers and announcements letters. He also has videos, including:

• E1050 Top 10 facts
• A S1024 first look
• Power10 infographics

For viewpoints outside of IBM, there’s Charles King and The Register.

Finally, bookmark this page on IBM Support. You can select the different models and easily determine which operating systems are supported, along with the minimum and recommended levels you should be running. Power10 is just the latest addition; you’ll find information on previous Power processors. I check this info all the time.

The new servers can provide quite a performance boost, particularly if you’re running on pre-POWER9 hardware. When you do choose to migrate, be prepared to be pleasantly surprised.

Originally published by TechChannel July 1, 2022

Rob McNelly looks at an upcoming IBM virtual event, an IBM Support update for AIX Toolbox open-source software and more

IBM is hosting a Power10 virtual event on Thursday, July 14, at 11 a.m. EDT. You can register here and you’ll find the event description below:

“Businesses are facing continued uncertainties. Resource availability. Volatility in demand and in costs. Now, more than ever, this requires flexible and reliable technology to deliver.

Join us for a virtual event to learn more about the latest from IBM Power. Hear from clients and IBM experts about how Power helps create digital advantage with hybrid cloud infrastructure to modernize, automate and secure your business with class-leading reliability.”

Just given the number of IBM Power Systems executives who are participating (click the link to see the list), it’s safe to assume that this is a big deal. Having taken part in a few NDA sessions regarding the upcoming announcement, I can’t get into specifics, but I think you’ll be very interested to learn about what’s ahead.

An Update on IBM Support for AIX Toolbox Open-Source Software

IBM recently announced that it is providing remote assistance with selected community-supported open-source products available through the AIX Toolbox repository. This is significant since support for AIX Toolbox open-source software wasn’t previously available through IBM AIX support cases. 

The service is formally known as IBM Support for Community Open-Software for AIX Toolbox. According to the announcement letter, IBM provides Level 1, Level 2, and Level 3 support for these AIX Toolbox packages: curl, db, dnf, dnf-plugins-core, expat, gettext, glib2, gnupg2, json-c, krb5, ncurses, openldap, python3, readline, sed, sqlite, texinfo, xz, zchunk, zlib, and zstd.

In addition, IBM will:

1. Provide remote assistance to the client for all covered products, through telephone from IBM’s support center, or electronically, in response to requests pertaining to the following:

• Basic, short duration installation, usage, and configuration questions for open-source packages running on AIX downloaded from AIX Toolbox only and code-related questions
• Diagnostic information review to assist in isolation of a problem cause; for example, assistance interpreting traces and dumps for installation and code-related problems
• For known defects, provide available corrective service information and information about obtaining a corrective fix from the AIX Toolbo

2. Assist the client in determining the cause of the problem and provide a corrective information-fix if it is available from the open-source community, AIX Toolbox or IBM Support

3. Report the defect to the open-source community and inform the client of known actions taken and the availability of the corrective information-fix if a new defect (referenced or without known correction) is identified

4. Provide the client with guidance on how to obtain patches from the AIX Toolbox, or maintenance updates or refreshes (collectively known as fixes) directly from the AIX Toolbox. IBM may provide workarounds as temporary solutions to the client or to the open-source community so the open-source community can create permanent patches. There is no guarantee that the workarounds will be accepted by the open-source community as part of the main code branch.

5. IBM will provide the security vulnerability fix, and when the security vulnerability fix is available in the community, will also provide support for all associated dependencies from the supported packages from the SPL downloaded from the AIX Toolbox repository The announcement letter goes into considerable detail, so take the time to read the whole thing.

What Constitutes a Senior AIX Admin?

Newsflash: Experienced IT people are hard to find. OK, you already knew that people leave their jobs. They take new jobs, they retire, what have you. Replacing workers is the focus of this IBM Community thread from March. 

Here’s a sampling:

“I’ve worked with admins which blew me away with the depth of their knowledge and their curiosity to know more! I always love it when they can teach me something too. I hang onto their names and network with them!

I’ve also had the opposite issue where system administrators were really just application administrators or other non-technical roles, where the system fell under their responsibility.

There are all types, but both will be listed in the same job title on their resume. […]

I think that a Senior is the one who has real experience and has not been doing just one thing, he is the one who is learning something all the time, he is the one who is trying to make things happen by doing something new.

But I think the most important characteristic of a Senior is that they are teachers, guides and mentors to others.

Of course, their experience allows them to avoid asking for help for little things, because of that they provide better results and in shorter times than others.”

It’s an interesting discussion, and a reminder that interesting conversations are happening on the Community pages.

The Importance of Keeping Current

This document explains why clients should keep their application, OS, and firmware up to date: 

IBM Remote Technical Support will recommend and encourage IBM clients to upgrade for these reasons:

• Adhere to best practices for keeping software levels in-sync
• Address known issues related to the reported problem
• Address undetermined or possible future issues using out-of-sync levels
• Take advantage of new enhancements or features 

IBM, like any vendor, is not and will not be capable of testing and verifying all combinations of applications, operating systems, and firmware levels. When considering all versions of all components, there are an infinite number of combinations to fully test all versions and combinations.

Remembering Gareth Coates

I want to take a moment to remember Gareth Coates, who passed away on June 15. I referenced his IBM blog and cited information he’d provided over the years. I learned quite a few tricks and tips from him.

Read and share memories and condolences here. 

Life is short, and none of us get out alive. Let those you love know that you love them.

Hiking Update 

For those keeping score, I successfully completed my North to South Rim Grand Canyon hike back in May, a few days after the North Rim opened for the season. It was hot, it was long, but it was beautiful and I cannot wait to do it again. Two days prior I fractured my toe, but luckily, ibuprofen was sufficient to allow me to get through it.

Just a reminder: Hiking in the Canyon during the summer is no joke. Sadly, a young woman recently lost her life three miles below the rim. 

Originally published by TechChannel April 20, 2022

IBM Champion Rob McNelly on HMC, VIOS, LPAR, AIX 7.2 and 7.3 tips, and more

It’s always nice to hear from readers. Even better is when I can confirm that my information is helping techies in the real world. My colleague Eric Hopkins recently put my invscout script to work for one of his clients. Here’s his story:

I have a customer who has smartly blocked outbound traffic to the internet from their NIM server, so I used details from your article to get a quick report of firmware status which would have taken me hours to gather and review manually.

Here’s how I did it in short order.
1. Downloaded ftp://ftp.software.ibm.com/software/server/firmware/catalog.mic to my workstation and used scp to get it to NIM
2. Used scp to distribute to a pile of VIOS. NOTE: I have SSH key exchange set up for ease of use, so I have $WCOLL defined as a host list file which contains #comments so I needed to exclude them.

            –START
            for i in `cat $WCOLL | grep -v \#`
            do
              scp /export/microcode/catalog.mic $i:/var/adm/invscout/
            done
            –END

3. Used dsh to run invscout
            # dsh /usr/sbin/invscout
4. Gathered the data from each host at /var/adm/invscout/<hostname>.mup using a fancy dsh command to exclude dshbak headers but leave invscout delimeters, but first ran a dsh command to list the files to make sure I wasn’t going to grab any old garbage
            # dsh “ls -l /var/adm/invscout/*.mup”
            # dsh “cat /var/adm/invscout/*.mup” | dshbak | egrep -v ‘HOST:|\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-‘ > VIOSmup

5. Used scp to get /export/microcode/VIOSmup to my workstation
6. Visited https://esupport.ibm.com/customercare/mds/mds and clicked on “Manual Upload” link.
7. Browsed my workstation for VIOSmup, selected radio button “Off line HTML” and clicked Big Blue’s “Upload file” button.

Moments later I was presented with the results which I printed to a PDF document for safe keeping.

Keep in mind, the IBM Support URL has changed yet again. Here’s the new one. If your script contains the old URL you will get an error.

This page provides additional detail. 

AIX 7.1: The Build Environment for AIX Toolbox Packages

As much as I like to believe we are all on current supported versions of AIX, I know that this is not actually the case. In case you missed it: 

We (AIX Toolbox team) are moving to AIX 7.1 as the build environment for AIX toolbox packages. Because of this, new versions of packages built on AIX 7.1 may not work on AIX 6.1 or earlier AIX releases.

Today our build environment for most of the packages is AIX 6.1. AIX 6.1 is already out of support, and we cannot continue this version forever. As a start we will just move the packages those we are upgrading to new version as per regular updates. The updates due to security vulnerabilities will continue to be built on AIX 6.1 (if vulnerable package was built on AIX 6.1) but in future we will move to AIX 7.1 for those packages as well.

This should not impact anyone who is using AIX 7.1 or later releases of AIX. The packages those are already built on AIX 6.1 will continue to be available on AIX toolbox repository. The AIX release name is part of package name where package was built. For example, python3-3.7.12-1.aix6.1.ppc.rpm is built on AIX 6.1 and can be installed on AIX 6.1 or later AIX releases whereas python3.9-3.9.6-1.aix7.1.ppc.rpm is built on AIX 7.1 and can be installed on AIX 7.1 or later releases.

This will also benefit in taking advantages of new features available in AIX 7.1 releases.

HMC Database Connectivity Fix and AIX 7.2 and 7.3 Installation Information

• Fellow IBM Power Champion Jaqui Lynch offers another helpful tip: “Unable to connect to Database on HMC virtual networks after updating from 3.1.0. to 3.1.3.” If you get this error when you click on virtual networks on the HMC after the update, the fix is here. 
• Along those lines, if you subscribe to IBM email notifications, you might have noticed more tips and tricks being shared there as well. Among those to recently hit my inbox: disabling unused ports on FC adapters; resolving LED 2700 when booting an AIX LPAR or PowerVM; and getting around an LPAR not booting due to error AA00E158 with HMC. While it’s nice to bookmark the actual links, information sometimes disappears from the internet. But for me, given how my memory works, if I run into one of these issues, having read about it previously usually rings a bell: I’ll at least recall seeing something. So when that happens, I can typically what I’m looking for—even if it’s not the doc I saw initially.
• Finally, I came across some installation tips for AIX 7.2 and 7.3

From Techie Twitter

• Chris Gibson retweeted this document on exporting VIOS backups on the HMC. Related: This information recently came in handy for an IBM i customer that was using VIOS but had no NIM server. 
• Take a few moments to familiarize yourself with IBM cloud-related backup strategies highlighted by Prarthana Rahul.
• And Kevin Adler cited my favorite new tool, the Power Linux Emulated Application Software Environment (best known by the easy to remember acronym PLEASE). The program is used to run “ppc64 Linux binaries on PASE for IBM i, which is an environment for running AIX binaries on IBM i.” 

Be sure to read the last paragraph in that GitHub link to fully understand how the whole thing works. It almost seems too good to be true.

Originally published by TechChannel March 18, 2022

IBM Champion Rob McNelly on the IBM Power Systems Ideas portal, a FLRT change and his latest outdoor adventure

I’ve written about the Request for Enhancement (RFE) process in the past, but recently, Brian Veale notified us of the new IBM Power Systems Ideas portal. This is a site where you can present your ideas about improving products and services and interact with IBM developers.

You’ll need an IBM ID to submit suggestions, comment and vote. But even without signing into the portal, you can view ideas by topic (AIX, HMC, VIOS, etc.) and filter through ideas that others have pitched based on their status (under review, future consideration, planned for future release, etc.) and the response to them (trending, popular). There are literally hundreds of ideas. Chances are you’ll like some more than others. By looking through existing ideas, you can avoid making the mistake of submitting an idea which has already been proposed. 

For what it’s worth, the most popular ideas at the time of this writing were:

• Dynamically editable min/max fields when an LPAR gets shut down
• Maintenance capabilities for vNIC failover
• HMC performance dashboard enhancements 

Some ideas that are not under consideration include more recent versions of WPARs, Ethernet drivers for running AIX 5.3 on POWER8 and redirecting commands on the HMC restricted shell. As is noted on the portal splash page: “Some ideas can be implemented at IBM, while others may not fit within the development plans for the product. In either case, the team will let you know as soon as possible. In some cases, we may be able to find alternatives for ideas which cannot be implemented in a reasonable time.”

Surely you have thoughts on what would make your favorite OS even better. Undoubtedly you have ideas about enhancements that could make your work days easier. Most admins I know have a ton of hard-earned experience, and they have no problem letting others know what they think. Of course not every idea magically becomes a reality, but I appreciate IBM’s willingness to allow their client base this degree of input. So if you’re not familiar with RFEs, check out the portal and learn. And if you’re already active in the community, the easiest way to contribute is to spread the word. Let others know about the new portal. Everyone who uses IBM solutions should participate in the RFE community.

An Important Update for FLRT Users

IBM Support recently noted that the Fix Level Recommendation Tool (FLRT) is now cloud based. This means that user scripts that call FLRT must be updated to this new URL. 

The change is explained in detail here. Note: “The functioning of this site will not be changed as a result of the move, but there will be a new URL. Initially there will be a redirect to the new site, which will be removed within a few months. There is no firm date set for the removal of the redirect.” 

While many of you are likely already aware of this, please help spread the word. If you’re unfamiliar with FLRT, read my primer. 

A Grand Canyon Update

For those who are interested, I’ll make it official: I survived my first Grand Canyon rim to rim hike. 

I was prepared, which made things somewhat easy. I discovered a nearby mountain has a “short” trail—a mile up, a mile down—that’s generally steeper and rockier than the Canyon trails. So I spent the past several Saturdays there, hiking up and back 5-8 times each day. That equates to 10-16 miles, with significant elevation changes.

This is a pretty good video of what to expect when attempting the rim to rim hike, which starts and ends at different points of the Canyon’s South Rim. For the record, I needed 10.5 hours to complete the hike, while the guy in the video got through it in just over 7.5 hours. Still, I’m pretty happy with how things went, and I’m looking forward to my next challenge. In May I’ll hike from the North to the South Rim. That’s a longer hike, and of course it will be much hotter by then, so wish me luck.

Originally published by TechChannel February 14, 2022

IBM Champion Rob McNelly on AIX migration tips and tricks, new Power10 performance information, IBM’s updated firmware release schedule and more

I’m regularly reminded of the benefits of IBM’s My Notifications email service. Yes, we all have too much traffic in our inboxes, but My Notifications is a flexible offering that provides worthwhile news and information. For instance, I subscribe to get alerts on recently released fixes. I also receive useful tips on topics like SAS disk array management:

How do I create, list and manage SAS Disk Array?
– General Help:
# sissasraidmgr -h
– Viewing the Disk Array Configuration:
# sissasraidmgr -Ll controller_name -j1
– Preparing Disks for Use in SAS Disk Arrays:
# sissasraidmgr -P -z disk_list
(For example, # sissasraidmgr -P -z hdisk1 hdisk2 pdisk3 pdisk4)
– Changing pdisks to hdisks:
# sissasraidmgr -U -z pdisk_list
– Creating a SAS Disk Array:
# sissasraidmgr -C -r raid level -s  stripe size -z pdisk_list
– Deleting a SAS Disk Array:
# sissasraidmgr -D -l controller name -d array_name
– Adding Disks to an Existing Disk Array:
# sissasraidmgr -A -l array name -z pdisk list
– Creating Hot Spare Disks:
# sissasraidmgr -H -z pdisk_list
– Deleting Hot Spare Disks:
# sissasraidmgr -I -z pdisk_list
– Displaying Rechargeable Battery Information:
# sissasraidmgr -M -o0 -l adapter_name
– Forcing a Rechargeable Battery Error:
# sissasraidmgr -M -o1 -l adapter_name
– Recovering from Disk Failures:
# sissasraidmgr -R -z pdisk_list
– Viewing the SAS device resource locations:
# sissasraidmgr -Z –o0 –j3 -l adapter_name
– Viewing the SAS device resource information:
# sissasraidmgr -Z –o1 –j3 -l adapter_name
– Viewing the SAS path information for the attached device:
# sissasraidmgr -T –o1 –j3 -l device_name
– Viewing the SAS path information graphically for the attached device:
# sissasraidmgr -T –o0 –j3 -l device_name

On a somewhat-related note, if you’re seeking AIX-specific news and information, Built On Power, a provider of solutions and services for Power Systems environments, is aggregating news and links on AIX and other topics. 

DNF and AIX 7.3 Migrations 

Chris Gibson has been tweeting about DNF, an install manager used in AIX migrations.

There’s an IBM Power Community blog devoted to DNF as well as an IBM Support page that lists potential errors when migrating to AIX 7.3:

“The rpm.rte 7.15.1.2004 installed on AIX 7.3 delivers 64-bit rpm. The DNF on AIX 7.1 and 7.2 used 32-bit rpm. You need to run the dnf_aixtoolbox.sh script again, to update to 64-bit DNF.”

Getting NTP Running on AIX 

A client was having trouble getting network time protocol (NTP) working on AIX. They would run lssrc –s xntpd and see it was inoperative. They would try to start it, and it would immediately die.

After running through the common issues like checking the config file, they called me. We read through more documentation before following one of the suggestions: run ps | grep ntp. Upon doing so, we learned that an xntpd process was running. After killing the process, we ran startsrc and the subsystem came right up.

For all of our technical know-how, sometimes diagnosing a problem requires little more than putting a fresh set of eyes on it. 

Virtual Serial Numbers Provide Flexibility for IBM i Clients

This seems very interesting for IBM i clients:

“IBM now offers you the ability to acquire a virtual serial number and assign it to a logical partition, also known as a virtual machine (VM). IBM i software can then be ordered on or transferred to the virtual serial number instead of being tied to a physical IBM Power serial number. Having the IBM i entitlement, keys, and support entitlement on the virtual serial number provides the flexibility to move a VM to a different Power machine.”

Whenever I’d bring up using Live Partition Mobility (LPM) with my IBM i friends, they would always remind me that many of their applications and licenses are tied to a physical serial number. This announcement will go a long way toward making LPM a reality for IBM i clients.

AIX 7.3 Certified for Oracle Database 19c; New IBM Redpaper; Power10 Quick Start Guide and System Firmware Releases

In December, Oracle announced the availability of the Oracle Database Release 19c with AIX 7.3. The press release points to this link, where users can confirm the supported combinations of IBM AIX 7.3 and POWER Systems (64-bit).

On Twitter, IBM Redbooks project leader Dino Quintero noted the release of a new Redbook that describes IBM Geographic Logical Volume Manager (GLVM) for data mirroring in Cloud deployments:

 “This publication is intended to help with the requirements to configure and implement GLVM   for cloud configurations. This book addresses topics for IT architects, IT specialists, sellers and anyone looking to implement and manage high availability and disaster recovery in the Cloud.”

Per Chris Gibson, IBM has published additional Power10 performance information. You’ll find a number of tips and tricks in this 15-slide presentation.

IBM’s firmware release schedule was updated on Jan. 31:

“In order to plan correctly for system firmware maintenance, there is a need to know when new Releases are coming out along with when a Release is planned to go end of service pack support. Therefore, the below table was created to provide this detail. Things can come up that alters this schedule somewhat, but this is the current plan. As things change, the table is updated to reflect status.”

Grand Plans 

For those of you interested in—or at least willing to tolerate—updates on my outdoor adventures, I have news. Next month I’ll be attempting my first one-day rim-to-rim hike of the Grand Canyon. I’ll actually start and finish from the South Rim, which is technically considered a rim-to-rim hike. Then in May I’ll spend a day hiking from the North to the South rim. Wish me luck.

Originally published by TechChannel January 21, 2022

IBM Champion Rob McNelly reflects on what he’s learned from readers—from mixed results updating VIOS on POWER7 to a cleaned-up invscout script

A customer was planning to patch their VIO servers on a POWER7 machine. You may recall my mention of another customer that had success running recent code on POWER7 despite the lack of documentation.

“I mention POWER7 specifically because a client with that system was wondering about their options. If you go to FLRT and plug in 8202-E4C for the machine type model, you’ll see that 2.2.6.5 is the latest recommended version of VIOS code. For an 8202-E4D model (POWER7+), the recommendation is 3.1.2.21.

What would you do in this situation? Is there enough of a difference between POWER7 and POWER7+ that you would hesitate to deviate from the recommendations? In this case, even when told by support to stand pat, the client was brave and chose to install 3.1.2.21 on their POWER7 box, because that version of VIOS is still supported. It worked. Of course every environment is unique, so your mileage may vary. And ultimately, getting to POWER9 should be your goal if it is at all possible. But I can report that at least one client is now successfully running 3.1.2.21 on POWER7.”

In this case, the customer’s 8202-E4C was running fine on VIOS 3.1.1.25. But with FLRT showing that 3.1.2.21 was supported on an 8202-E4D, they decided to roll the dice and move to 3.1.3.14. It didn’t go well. Their NPIV mappings stopped working, and attempts to login to the SAN switch triggered errors from the HMC lsnportlogin command. Using the chnportlogin command didn’t work either. Here’s what they saw:

wwpn_status=2,logged_in=unknown,wwpn_status_reason=invalid error number

Lacking time in their change window for additional testing, they ended up restoring from a mksysb and returning to 3.1.1.25. Everything worked again. Now the plan is to stay there until the system is retired. Naturally this setup isn’t supported by IBM, but the customer feels it’s their best option. Incidentally, the customer is running another 8202-E4C box at VIOS at level 3.1.2.21 with no issues.

So there’s hope, but also a strong dose of reality. Yes, some of these older machines can be updated beyond the stated supported levels, but you’ll need to do some testing (and, obviously, capture a mksysb, run viosbr, etc., even before that. As I recently said on Twitter, nobody cares about backups, but they do care about restores). Realistically though, it appears we’ve about reached the limits of extending VIOS on POWER7 machines.

Still, I’d love to hear more about this topic, particularly if you’ve been able to get even later VIOS versions to work on POWER7 hardware.

Simplified invscout Script 

I greatly appreciate reader feedback, because typically I learn something. Here’s another recent reader interaction about a script that wasn’t working.

You may have seen my TechChannel video about invscout. I had linked to a script that’s supposed to automate the download of the catalog.mic file, copy it to multiple LPARs, consolidate the output and send it to IBM. However, a reader told me the script did not work as written. Fortunately, it was a quick fix. I changed a couple URLs and did other small simplifications that the reader requested.

Obviously you’ll need a machine that can access the LPARs you select to run invscout, and of course you’ll need internet access to automate the catalog.mic file download from IBM and send the file for processing. Depending on restrictions in your environment, it may take more manual steps to get this to work, but the general idea is that you start with a device—a NIM server or something similar—that has ssh keys set up to log into LPARs.

Be sure to edit the /usr/local/etc/servers file (or change the location to something you would prefer) and verify the TEMP directory you plan to use for this. You’ll also need scp, curl and wget installed.

The original script is capable of self-cleanup, so you may want to automate that by adding those sections in as well. The original also emails the output. I added a couple of comments to highlight where the IBM URLs had changed. You may prefer running the original script with my URL changes, but I also wanted to provide another, shorter version. Given the frequency with which useful tech stuff disappears from the internet, it never hurts to double up on these things.

#!/bin/ksh
# script: generate_survey.ksh
# purpose: To generate a microcode survey html file
# where is my list of servers located?
SERVERS=/usr/local/etc/servers
# what temporary folder will I use?
TEMP=/tmp/mup
# what is the invscout folder
INV=/var/adm/invscout
# what is the catalog.mic file location for invscout?
MIC=${INV}/microcode/catalog.mic
# user check
USER=`whoami`
if [ "$USER" != "root" ];
then
    echo "Only root can run this script."
    exit 1;
fi
 
# create a temporary directory
mkdir $TEMP 2>/dev/null
cd $TEMP
 
# this url had changed from the original script
wget ftp://ftp.software.ibm.com/software/server/firmware/catalog.mic
 
# move the catalog.mic file to this servers invscout directory
mv $TEMP/catalog.mic $MIC
 
# distribute this file to all other hosts
for server in `cat $SERVERS` ; do
   echo "${server}"
   scp -p $MIC $server:$MIC
done
 
# run invscout on all these hosts
# this will create a hostname.mup file
for server in `cat $SERVERS` ; do
   echo "${server}"
   ssh $server invscout
done
 
# collect the hostname.mup files
for server in `cat $SERVERS` ; do
   echo "${server}"
   scp -p $server:$INV/*.mup $TEMP
done
 
# concatenate all hostname.mup files to one file
cat ${TEMP}/*mup > ${TEMP}/muppet.$$
 
# Sometimes, the IBM website will respond with an
# "Expectation Failed" error message. Loop the curl command until
# we get valid output.
 
stop="false"
 
while [ $stop = "false" ] ; do
# The following url had changed from the original script. I also had to make sure that it was all on the #same line, it is all one command.
curl -H Expect: -F mdsData=@${TEMP}/muppet.$$ -F sendfile="Upload file" http://www14.software.ibm.com/support/customercare/mds/mds> ${TEMP}/survey.html
 
#
# Test if we see Expectation Failed in the output
 
#
 
unset mytest
mytest=`grep "Expectation Failed" ${TEMP}/survey.html`
 
if [ -z "${mytest}" ] ; then
        stop="true"
fi
sleep 10
done

At this point I moved the survey.html file onto a machine with a browser, and it gave me a very useful report. Chris Gibson has some great examples of the reports and information you can expect to see. Scroll down for specifics about adapter firmware updates. The red type really pops, and once you’ve updated your firmware and rerun the report, seeing the red disappear can be very satisfying as well.

Originally published by TechChannel January 4, 2022

IBM Champion Rob McNelly on the Log4j security vulnerability, helpful AIX tips and tools, time drifts on POWER8 and POWER9 servers, and more

System security is no laughing matter. Well, occasionally it’s a laughing matter, but I’ll get to that in a moment. On Dec. 15 IBM issued a security bulletin regarding a vulnerability in Apache Log4j that affects the Power HMC. Check the IBM PSIRT blog for the latest updates: 

“… Product teams are releasing remediations for Log4j 2.x CVE-2021-44228 as fast as possible, moving to the latest version that’s available when they are developing a fix. Where possible, the dependency on Log4j is removed entirely.

IBM is aware of additional, recently disclosed vulnerabilities in Apache Log4j, tracked under CVE-2021-45105 and CVE-2021-45046. Work continues to mitigate or remediate these vulnerabilities in products and services that already have released a remediation based on Log4j 2.15.

With so much active industry research on Log4j, mitigation and remediation recommendations will evolve. We are actively assessing the latest Log4j developments and will share updates accordingly.”

The blog also lists products that are confirmed not impacted, as well as products that have been remediated. Patch your systems as soon as possible.

Now for a little Log4j levity. In a reprise of the great jif/gif debates, admins are again arguing about pronunciation. Is it “log four jay”? “Log forge”? I’ve seen and heard both. Twitter has also weighed in on this very important matter:

            Problem: Apache Log4j
            Solution: A patchy Log4j
            *crowd booing*

Encrypting AIX Logical Volumes

On Twitter, IBM’s Soumya Menon cites documentation about encrypting logical volumes in AIX:

“Starting with IBM AIX 7.2 with Technology Level 5, the Logical Volume Manager (LVM) supports the data encryption at the logical volume (LV) level. Using this feature, you can encrypt the data at rest to protect data exposure because of lost or stolen hard disk drives or because of inappropriately decommissioned computers. The term data at rest refers to an inactive data that is stored physically in any digital form.

Each LV is encrypted with a unique key. The logical volume data is encrypted before the data is written to the physical volume. This data is decrypted when it is read from the physical volume. By default, data encryption is not enabled in logical volumes. You must enable the data encryption option at the volume group level before you enable the data encryption option at the logical volume level.

The hdcryptmgr command manages the encryption keys, data encryption, and data decryption of the logical volume.”

An Option for Decoding and Summarizing AIX I/O Error Messages 

Here’s an interesting tool for your bag of tricks:

            NAME
            summ
            PURPOSE
            Diagnostic tool for decoding and summarizing AIX I/O error messages
            SYNTAX
            summ [Flags] [Filename]
            FLAGS
            -e Include FC driver error numbers for each error.
            -p Paginate the output.
            -r Reverse the order of output.
            -s Include sequence numbers in each line’s header.
            -c I/O retry cmd_history failure time and reason
            NOTE: summ –help displays the flag options.

DESCRIPTION
The summ command is an AIX only diagnostic tool used to decode Fibre Channel and SCSI disk AIX error report entries. It is an invaluable tool that can aid in diagnosing storage array or SAN fabric-related problems providing the source of the error.

The script generates single-line error messages enhancing the readability of the AIX error report. The tool is used by IBM Support worldwide, and is considered safe to run in a production environment.

Timeout Issues When Querying VIOS Resources From the HMC 

IBM Support has an explanation:

Question: When trying to query virtual resources configuration on a managed system from HMC, it may happen that it takes very long time before completing or it fails with a timeout error message.

Cause: Any time a query is performed from HMC, a call is made to all VIOS on the managed system to get details on the configuration. On VIOS, the vio_daemon will proceed with this request by sending a query to the CMDB and respond to HMC.

There are different possible issue which could lead in timeout or at least long delay for this query, and the most common error message seen is:

-> The system is currently too busy to complete the specified request. Please retry the operation at a later time. If the operation continues to fail, check the error log to see if the filesystem is full.

Answer: The error above let us think that the VIOS is currently suffering some performance issue. Indeed the VIOS has to manage all the resource shared to client lpar (including disk access/IO, network communication…), but it also has to deal with all resource management request from all connected HMC (and in some case Novalink, PowerVC or other management product).

To reduce the risk of experiencing timeout issues, use the part command to monitor VIOS resources (CPU/memory). 

Time Drift on POWER8 and POWER9 Servers 

Also from IBM Support:

“Some clients noticed that the Power server time drifts seconds per day when compared to other systems, wall clock, or an NTP reference. They might observe it over a period of days, weeks, or months. The client can be asking questions.

Why is the server behaving differently?
Is there something wrong with my server?
Why must I use NTP when I never had to before?
Why is IBM not told me they changed the TOD accuracy of the server?

It does not indicate hardware needing replacement. The immediate suggestion is that clients use Simple Network Time Protocol (SNTP)/NTP as the power system Time of Day (TOD) can be expected to drift seconds per day when NTP synchronization is not used.

The only sure method to eliminate TOD drift is to deploy NTP and it is the IBM-recommended method to synchronize partition and system time and date for several generations of Power servers. Configuring and deploying NTP is outside the scope of the document but is described by OS documentation in IBM Knowledge Center. The reference section has pointers to some of the documentation.”

For more on time drift, see my old post from 2009.

E1080 at a Glance, AIX 7.3 Released, a Handy Tool for Job Hunters, NIM Install Troubleshooting 

• Nigel Griffiths posted a pair of images that summarize key features of the IBM Power Systems server model E1080. Download them from IBM Support.
• We had the open beta, now we have the real thing, AIX 7.3 has been released. Keep in mind the requirements: an IBM POWER8, POWER9, POWER10, or later, technology-based server. Also note that POWER8 Nutanix (CS821 and CS822) does not support AIX 7.3.
• According to his Twitter bio, Ron Gould is a systems and network administrator. He’s come up with an interesting tool to help other techies update and customize their resumes. Download the relevant files here. 
• If you’ve ever had trouble installing NIM, bookmark this page from IBM Support. 

Note: “This document is intended as a reference guide for troubleshooting common NIM LED hangs. It is not intended as a fail-safe resolution guide, however [these] steps represent the most likely causes and resolutions to various NIM hangs.”

Edit: I am looking forward to the next half marathon at Lake Powell

Originally published by TechChannel November 29, 2021

Dynamic LPAR is now available for SAP HANA databases. Get details on this, along with updated Power10 and AIX 7.3 documentation and more from IBM Champion Rob McNelly.

Dynamic LPAR is now available for SAP HANA databases.

The new capabilities allow HANA admins to adjust memory on active LPARs without shutting down partitions. This tweet has more information,including an important reminder about the potential need to run dynamic platform optimizer (DPO) should you experience performance degradation.

This is just another reason to consider running SAP HANA on Power Systems servers if you aren’t already.

“With IBM POWER9 processors and IBM PowerVM, Power Systems can host up to 16 production SAP HANA databases on a single server. You can granularly allocate memory and cores across SAP HANA instances to meet precise capacity needs. Support for shared processor pools lets you dynamically distribute compute capacity across SAP environments, reducing total cost of ownership (TCO). On-demand workload scaling allows you to quickly and easily add more cores and memory to SAP HANA workloads—without configuration recertification by SAP.

“This flexible solution allows you to run both SAP and non-SAP applications—including transactional, analytical, memory-intensive, and I/O-intensive workloads—on a single platform. You can also run legacy SAP applications alongside SAP S/4HANA workloads and migrate at your own pace. Flexible resource allocation lets you support short-term, long-term, and shifting demand.”

Service and Support Best Practices Docs Updated 

Chris Gibson shared this link, which lists numerous documents that recommend service and support strategies for IBM systems and software. Recently updated AIX docs include Power10 Performance Best Practices and POWER9 Performance Best Practices. Scroll down to the hardware and firmware section and you’ll find the POWER8, POWER9 and Power10 system firmware release planned schedule, which was also just updated.

Note: These brief checklists and docs should not be confused with the much more comprehensive Power Implementation Quality Standard document that was just updated to version 2.5. The newly updated pages are clearly marked as you make your way through the presentation.

AIX 7.3: A Technical Review 

Chris also let me know about the Singapore AIX/IBM i/Linux on Power Meetup Group. The next monthly meeting is Nov. 26. He and Anthony Steel will discuss new features in AIX 7.3 (as well as 7.2). The presentation will last about an hour. A Q&A session will follow.

If you’re coming by this information after the fact, check out the meeting archives. 

Power10 Presentations and Videos 

Speaking of user groups, and the latest announcement information, some interesting replays are available. The UK-based Power VUG Technical Webinar Series hosted these recent presentations:

• Session 110: PowerVM features in Power10 systems and HMC V10, CMC, Enterprise Pools 2.0 Enhancements
• Session 109: Green is easy, with IBM Power—how Power10 contributes to your Sustainability journey, while saving cost at the same time

And check out these Power Systems Virtual User Group webcast replays:

• AIX 7.3 Announce with Maria Ward, Carl Burnett & Brian Veale (presentation and video replay) 
• IBM Power10 Announce with William Starke & Joe Armstrong (Power10 processor presentation, Power E1080 presentation and video replay)

Cool Twitter-Adjacent Stuff

On Twitter, Kiran Tripathi cites options for capturing and exporting VM instances, here.

“You can capture and export an AIX or IBM i VM instance by using the Power Systems Virtual Server user interface or CLI. A VM is captured as a volume backed image. The image is stored in new volumes on the storage providers. An image can be exported to an IBM Cloud Object Storage (Cloud Object Storage) bucket. When an image is exported, the volumes of the image are copied and packaged in an Open Virtualization Appliance (OVA) file. The OVA file is compressed by using gzip before it gets uploaded to the IBM Cloud Object Storage bucket.

“When you capture and export a VM, you can choose the image catalog, COS, or both as destinations. The image catalog resides on the IBM Power storage area network (SAN). IBM’s COS is encrypted and dispersed across multiple geographic locations, and accessed over HTTP by using a REST API. This service uses the distributed storage technologies that are provided by the IBM COS System (formerly Cleversafe). You can always export your image in your image catalog to COS at a later point. You can also deploy the captured image to create a clone of the VM by using a different network configuration.”

In a post on LinkedIn, Chris Peterson explains how AIX APIs can be used to explore user password histories:

“This one is a bit specialized and not something everyone should meddle with. Of course, your code has to run as root in order to query this “database” in the first place, so all’s fair in love and business.
…

“I was extremely pleased by the announcement that AIX’s next release will default to a much better password hashing algorithm than crypt() that allows for longer passwords! Huzzah!
…

“Yet another—mostly documented—API that sets AIX apart from some of the “competition.” If you know what to look for and have an idea where to look, you’re almost guaranteed to find ways to make AIX one of the most secure and auditable platforms anywhere.”

Tell Yourself You Can, and You Will 

When I heard that a friend was going to run a half marathon, my first thought was: “There’s no way I could do that.”

But the thing is, I’m an experienced runner. I love running. Sure I’d never run that distance before, but was it really that difficult? I decided to find out. One day I extended one of my regular runs to see how far I could get. A half marathon is 13.1 miles; I ended up covering 14 miles with relative ease. So yes, I realized, I can run a half marathon—and I did just that at the recent event at Lake Powell in northern Arizona.

If you tell yourself you can’t, you’ll be right. But if you tell yourself you can, you’ll also be right. It may take time, effort and practice. You may fail once or multiple times when trying something new, but that’s OK. You’ll most certainly fail if you don’t try. It’s far better to fail at something than succeed at nothing.

Edit: did you install the open beta or did you go straight to the GA release?

Originally published by TechChannel October 14, 2021

IBM Champion Rob McNelly on his new TechChannel AIX video series, and the latest news on Power10 and AIX open beta

A quick professional note: I’ve launched a video series for TechChannel. I’ve recorded three so far, including my latest on the Technical Support Appliance (TSA). Take a moment to check them out—the videos all clock in at around four minutes—and then send me suggestions for topics you’d like to see covered going forward. Being new to video creation, it’s a challenge, but I like presenting this information in a new way. Note that each video is transcribed, so read the text if you prefer that to me talking.

The Latest on the AIX Open Beta and Other Power10 Topics

Have you been reading the IBM Power Community blog? I’ve been enjoying the content, including this entry on PowerVM features in Power10 Servers. PowerVM components such as VIOS, the HMC and server firmware are updated in the new release. The piece also covers discontinued features such as active memory sharing and workload management groups.

Here’s another good entry: “HMC V10 R1 M1010 Features and CMC & Enterprise Pools 2.0 Enhancements”: 

“As IBM introduced the E1080 system, the first in a generation of servers based on the Power10 processor, comes a newer of version HMC enabling management of Power10 Systems. HMC V10 comes with a lot of new features & enhancements which will enable a seamless systems management experience and addressing many Request for Enhancements (RFEs).”

Also be sure to read this update on the AIX open beta:

“On behalf of the whole AIX team, I am excited to announce that the AIX 7.3 Open Beta is now LIVE! The free AIX 7.3 Open Beta provides an early view of the new AIX 7.3 release that will be available later this year. The Open Beta is open to anyone that would like to participate, with multiple options for getting access to the early release.

· Take advantage of $1500 in PowerVS credits to install off-prem via IBM Power Virtual Server.
· Install the early code on your own Power8 or later server.
· Software Vendors that need access to a server can get special access thru the IBM Technology Zone.

This is a unique opportunity to test out AIX 7.3 ahead of its release! Participants will be able to take advantage of some of the new features and enhancements, including:

· Increased file and filesystem sizes.
· Python and Bash frameworks running directly with AIX.
· Support for dnf command for installation of open source packages from the AIX Toolbox.
· Reduced time to dynamically add processors/memory to a running LPAR.
· Reduced IPL times for multi-terabyte memory LPARs.
· pigz and zlibNX commands now transparently use NX GZIP acceleration on Power9 and Power10
enhanced support for logical volume (LV) encryption to include rootvg and dump device.
· tcp protocol stack supports CUBIC.
· Additional IP Security (IPsec) enhancements.
· Create an OVA file from a mksysb using create_ova command.
· Create ISO image from mksysb_iso command.
· Integration with new IBM Open XL C/C++ and Fortran for AIX 17.1 compilers.”

I have downloaded the code and started testing in the lab. Enjoyable as this is for me, these experiences always make me think about how much has changed with the OS, even since AIX 7. Of course compared to AIX 6, which I loaded on VIOS back in 2007, it’s a whole new world now.

As you get hands-on with AIX 7.3, you should spend time in the user forums. I set up my subscription to receive emails with comments, and beta participants are already providing great feedback and information. Incidentally, the beta runs till the end of January, so you can still join the party. Register here.

Mapping VIOS and Virtual FC Adapters

Chris Gibson just posted a short but timely piece on mapping VIOS and virtual FC adapters in Power10 environments. 

The upshot is that you can now get output with the client adapter name, vios partition, vfc host adapter name and location, physical adapter on VIOS, etc. This makes mapping the adapter on your VIO client out to the vio server and actual physical adapter and port that much easier.

vpgadmin Explained

A reader was looking at /etc/passwd on the VIO server and noticed vpgadmin. What does vpgadmin do? IBM Support has the answer. 

“What is the purpose of User ID “vpgadmin” on the Virtual I/O Server? Can it be removed? Can the user settings be modified?

Cause: The user ID is being flagged during a security scan?
Answer: The vpgadmin is a new user introduced in version 2.2.6.31 and above, as well as in version 3.1. Note: If the Virtual I/O Server is running at ioslevel lower than 2.2.6.31, and it is later updated to that version or higher, the vpgadmin ID will be automatically created during the update process.
vpgadmin is used internally and required for Virtual I/O Server database administration (SolidDB and Postgre DB).

This user should not be removed or disabled since it is reserved for internal VIOS administration. Doing so will lead to unpredictable results and will leave the Virtual I/O Server in an unsupported configuration. Changing the user’s group ownership is not supported.”

There’s lots more, so read the whole thing on the IBM Support site.

Cloud Backup Management Options

If you’re considering a cloud migration, you may be interested in the contents of the new IBM Redpaper, “Cloud Backup Management with PowerHA SystemMirror.” The document shows you how to take a backup of your data by either cloud backup or remote storage. It also explains how to recover your data if there is a disaster. At 11 pages, it’s a quick and valuable read.

Edit: A nice roundup of links

Originally published by TechChannel September 14, 2021

IBM Champion Rob McNelly on Power10 E1080 server highlights, and resources you can use to learn more

On Sept. 8, IBM announced the new Power10 server, the E1080 (9080-HEX). While much has already been written about the new server, I’m here to share more details. If you take the time to check out even a few of the many links in this article, you’ll soon be up to speed. I’ve also included links to in-depth training, which will be available very soon.

E1080 Facts and Figures 

Let’s start with IBM’s updated server facts and features and this new Redbook. You should also watch Nigel Griffiths’s “fast facts” and “10 highlights” videos. Both come in at under 15 minutes.

As you can imagine, Griffiths is a busy guy these days. He’s also doing a live presentation, “Power10 from the Hands-on Experience,” on Sept. 15. He’ll essentially take a new system apart and show you what’s inside. In other words, you’ll get the view that is usually restricted to the CEs who set up and service systems.

For more in-depth information, check out the Sept. 30 Power Systems Virtual User Group presentation, “IBM Power10 Announce,” with Bill Starke and Joe Armstrong. Register here.

Of course you’ll want to explore what IBM is saying about Power10. Start with this easy to remember landing page, this new server data sheet, and this whitepaper. Also check out the Power10 AR experience and interactive demo. Finally, there’s this interesting blog post and this explainer about the significance of a new SAP benchmark: 

“Today, SAP published a new SD 2-tier result for IBM’s soon to be announced Power E1080. First the highlights:

174,000 SD Users
955,050 SAPS
120 cores

Wait, almost 1M SAPS with only 120 cores? HPE achieved 670,830 SAPS (122,300 users) with 224 cores on their Superdome Flex 280 with the Intel Xeon Platinum 8380H Processor in January 2021.

This new result is almost 3 times the SAPS/core of HPE’s biggest and baddest system. (Funny note: autocorrect tried to change “baddest” to “saddest.”) This new result is also about 33% faster, on a per core basis, than the previous Power 980 result published at the end of 2018. That is certainly not remarkable since Intel’s per core performance on this benchmark also increased about 69.5%, since 2017 … sorry, missed the decimal, 0.695%. (Comparing two Dell 2-socket results, Intel 8180 & Intel 8380).”

Administrative Changes

AIX admins will notice something new on Power10: It’s much easier to keep track of the status of your software maintenance agreements (SWMA). You’ll find notifications about the expiration of SWMAs in both the error log and on the HMC: 

“With the introduction of the Power10 server, IBM expands upon the use of the update access key (UAK) with the addition of an SWMA UAK. This functionality provides proactive notification of AIX SWMA expirations to ensure continued and uninterrupted software support. AIX SWMA UAKs do not limit the operability of or capability to update AIX.”

You should also be aware of changes with the UAKs: 

“What do I need to know about AIX Update Access Keys and Power10?

IBM Power10, or later, processor-based servers include technology that helps to manage and keep your SWMAs current so that you can apply AIX updates and receive support from IBM. The server checks for an active SWMA when updating the AIX operating system. The server utilizes an AIX UAK that includes the expiration date for the associated SWMA. Informational messages are generated when the release date for the AIX operating system has passed the expiration date of the AIX UAK and during normal operation. Additionally, the server periodically checks and informs the administrator about AIX UAKs that are about to expire, AIX UAKs that have expired, or AIX UAKs that are missing. It is recommended that you replace the AIX UAK within 30 days prior to expiration.”

Of course you’ll need to run new HMC code to go with your new Power10 server. Now you can get the new virtual appliance or update the code on your HMC hardware, assuming it’s compatible with the new software version. 

Also in This Announcement 

Along with enhancements to PowerVM and PowerVC, PowerSC 2.0 is now available: 

“IBM PowerSC 2.0 strengthens and simplifies system security management by combining the capabilities of IBM PowerSC Standard Edition 1.3 and IBM PowerSC MFA 1.3 and adding a major new feature:

Endpoint detection and response (EDR) capability, which includes the following:
Host-based intrusion detection system (HIDS). HIDS analyzes the traffic to and from a specific computer for signs of possible intrusion incidents, violations, or imminent threats. HIDS can also monitor key system files and any attempt to overwrite these files, a function that works in conjunction with file integrity monitoring (FIM). HIDS can do log-based intrusion detection, which overlaps with log inspection functionality, and includes time-based alerting and active response.

Log inspection and analysis. PowerSC 2.0 can identify important security events buried in the OS and application logs and ignore relatively unimportant events.

Event context and filtering. Events are sorted, prioritized, filtered, categorized, and put in context to help identify anomalous activity.

Incident response. This methodology, used to respond to and manage a cyberattack, aims to reduce the damage and help recover as quickly as possible.”

Finally, in case you were wondering, Power10 E1080 servers have a PVU of 120. 

There’s a lot here, and yet I can’t say I’ve covered everything. IBM never sleeps, after all. So feel free to reach out if there’s something I’ve missed.

Edit: I find it so much faster to make these changes from the command line vs the GUI

Originally published by TechChannel August 23, 2021

Rob McNelly highlights resources you can use to learn how to create an LPAR without using the HMC GUI, and use the CLI for dynamic LPAR operations

Gareth Coates, whose work I’ve cited previously, recently explained how to create an LPAR without using the HMC GUI:

“Here is a suite of files to create an LPAR on an IBM [Power Systems server]…. There are shell scripts and configuration files.

You need to make changes to suit your particular environment. I’d say that the chance of them working on your systems without such modifications is nil.

A tar file is available for download in the “Attachment” section at the bottom of this page.

My original motivation for developing the suite was when I was working on a particular project. To access the HMCs from my desktop, network packets had to go through several firewalls using a complicated WAN. There were many LPARs on numerous Enterprise Class systems and many users of the HMCs’ GUIs. The HMCs were busy and paging was happening. The result was a slow network connection; both low bandwidth and more importantly, high latency! It could easily take over 5 minutes to fill in the HMC pages and create the LPAR, even when all of the properties were already defined and I just had to complete the forms. Then, I had to use DLPAR to add virtual adapters to the VIO [servers] and edit the VIOS profiles.

Using “malt,” I could achieve all of the steps in under 30 seconds. Sometimes, I was tasked with creating several (maybe 10) LPARs with similar configurations but with different LPARid numbers, you can imagine the time which the scripting saved.

I found that improvements to the HMC and new powerful products like PowerVC and the “LPAR Provisioning Toolkit” took over from malt, so I did not post my utility. Recently, several customers have asked for access to it, so here it is.

I understand that it has some limitations, but it is what it is. I am not doing any more development on it. If people extend it, for example to use SSP storage, and let me have the code, I’ll consider posting it here …”

Be sure to read the entire support doc for complete instructions.

Getting the Most From Dynamic LPAR

Along those lines, my coworker Eric Hopkins recently helped a client better understand how to use the command line interface (CLI) for dynamic LPAR (DLPAR) operations:

“The full capabilities to dynamically add or delete devices or edit the contents of a profile using CLI of the HMC still exists and has become the most direct method for implementing those changes. While confusing at first glance, the CLI offers a logical repeatable method of DLPAR actions.

The following examples will define the needed information for each task and utilizes a defined variable within the command structure. A real-world example will also be shown for clarity and syntax.

Two commands are needed to completely add an operational device. One command to add it to the LPAR that points to the managing VIOS, and one command to add it to the VIOS that points to the LPAR.

The syntax of the action being taken must be understood with reference to where the intention of the command is directed to add or remove a device. If a virtual fibre device is being added to an IBM i or AIX LPAR the $VIRTUAL_SLOT is the Adapter ID of the virtual device on that LPAR, while $REMOTE_SLOT refers to the managing VIOS for that device. Similarly, if this virtual fibre device is being added to the VIOS, $VIRTUAL_SLOT refers to the Adapter ID of the virtual device on that VIOS while $REMOTE_SLOT refers to the Adapter ID of the virtual device on the corresponding LPAR.

Best practice is that this number matches. However, if it doesn’t match, it’s important to know the distinction between the two variables.

Once the command has been accurately issued, the device will appear in the running operating system. At this time it is imperative to save the running config to the profile of the LPAR or VIOS. Without this step, when the LPAR is halted and started from its profile, the device will no longer exist.

Variable                                  Information needed:
$MACHINE_NAME              Server-9009-22A-SN8675309
$VIOS_NAME                        VIOS1
$LPAR_NAME                       nim01
$VIRTUAL_SLOT                  315
$REMOTE_SLOT                  315
$DEVICE TYPE                      fc (fibre), eth (ethernet), scsi (SCSI)
$ADD_OR_REMOVE            a (add), r (remove)
$SERVER_OR_CLIENT        server (Adding on VIOS), client (Adding on LPAR)

Adding a virtual fibre channel device to an LPAR:

chhwres -r virtualio -m $MACHINE_NAME -o $ADD_OR_REMOVE -p $VIOS_NAME --rsubtype $DEVICE TYPE -s $VIRTUAL_SLOT -a "adapter_type=$SERVER_OR_CLIENT,remote_lpar_name=$LPAR_NAME,remote_slot_num=$REMOTE_SLOT"
 
chhwres -r virtualio -m Server-9009-22A-SN8675309 -o a -p VIOS1 --rsubtype fc -s 315 -a "adapter_type=client,remote_lpar_name=nim01,remote_slot_num=315"

Adding a virtual fibre channel device to a VIO:

chhwres -r virtualio -m $MACHINE_NAME -o $ADD_OR_REMOVE -p $VIOS_NAME --rsubtype $DEVICE TYPE -s $VIRTUAL_SLOT -a "adapter_type=$SERVER_OR_CLIENT,remote_lpar_name=$LPAR_NAME,remote_slot_num=$REMOTE_SLOT"
 
chhwres -r virtualio -m Server-9009-22A-SN8675309 -o a -p VIOS1 --rsubtype fc -s 315 -a "adapter_type=server,remote_lpar_name=nim01,remote_slot_num=315"

Removing a virtual adapter in slot 315 from an LPAR or VIOS:

chhwres -r virtualio -m $MACHINE_NAME -o $ADD_OR_REMOVE -p $LPAR_NAME -s $VIRTUAL_SLOT
 
chhwres -r virtualio -m Server-9009-22A-SN8675309 -o r -p nim01 -s 315

* Don’t forget to save running config!

1. On the HMC GUI check the box by the LPAR name.
2. From the ACTIONS pulldown menu choose Profiles, then Save Current Configuration.
3. A window will pop up to allow overwrite of the existing profile or the option to save to a new profile.”

Everyone should be aware of these handy options. Even in smaller, static environments where the GUI is typically sufficient, these alternatives can be handy when installing systems or applying changes.

Edit: I still wonder about keeping a spare water heater on hand

Originally published by TechChannel July 30, 2021

Rob McNelly on VIOS admin tips, OpenSSH for AIX, effective support and more

Recently my water heater died. Fortunately it’s relatively new and still under warranty, so I didn’t have to dip into my pockets to get it repaired. But I did have to wait, because the manufacturer needed to ship a part, which took a few days.  

Supply chains are still being stretched, so it wasn’t surprising to learn that the required part wasn’t available in my area. Since I live in Phoenix, having to take a few showers in lukewarm water isn’t the worst thing this time of year, but still, what’s the point of a manufacturer’s warranty if my issue can’t be promptly resolved?

With all the camping I do, I’ve collected quite a bit of equipment over the years. A lot of this gear, like cots and sleeping bags and air mattresses, serves to keep me warm and comfortable. Given that I also have a 12-volt fridge with sufficient batteries to run it, not to mention a vehicle with a generator and an AC unit, one could argue that I’m more of a “glamper.” But having options, based on where you’re going and what you’re doing, is important. My point though is that during this recent stretch, I considered deploying one of my favorite camping comforts at home: a portable tank-less water heater.

While I don’t have this exact system, you get the idea. It’s great when you want to clean up at the end of a hot, sweaty, dirty day in the back country, but it doesn’t have much use in the house. It’s not an option to warm water for the dishwasher or washing machine. And as I said, with Arizona summers being what they are, I can get by without hot showers for now.

I must admit, even though it was just a few days, I missed the luxury of instant hot water. I’m sure my grandparents wouldn’t skip a beat without the convenience of an on-demand, whole house water heater. But me, the rugged outdoorsman? I was reminded how soft I am living in this modern world.

Anyway, to put a tech spin on this, I liken it to having a bares-bones disaster recovery solution. My little water heater is like a small backup box that will get me by in a pinch. By comparison, my regular home water heater is a powerful production machine that gives me everything I need to handle my water workload.

Would it make sense to purchase a second full size backup water heater, just in case I needed it? No. After all I’m not using my hot water to make money or service my customers, and it’s not as if I could lose millions for every minute without hot water. But the calculation is the same. What’s the cost of downtime? How valuable is continuous availability?

The part eventually arrived. In hindsight, it may have been faster had the manufacturer simply replaced my broken water heater. Of course from their perspective, shipping a single part is easier and cheaper. So I guess I come away from this episode with even greater appreciation for IBM support. Because when I need computer parts for a server, IBM generally ships and installs them promptly.  Another advantage is that more times than not, my server will proactively call home. On occasion it’s actually IBM that lets me know when a part is failing and must be replaced. I sure wish my water heater could call home. 

OpenSSH for AIX: What’s New 

Chris Gibson has a nice write-up about the new features available with the latest version of OpenSSH for AIX:

The latest version of OpenSSH for AIX 7.2 now supports POWER9 HW GZIP! The requirements for this new capability are as follows:

• Processor mode: Power9 (instead of Power9_Base)*
• AIX Level: 7.2 TL 4 and above*
• zlibNX: 7.2.4.0 NX accelerated zlib compression library*
• OpenSSH version: 8.1.102.2102 and above*
• OpenSSL version: 1.0.2.2100 and above *
• ssh commands (like ssh, scp etc) should be run with -C option (compression enabled) 

Read the whole thing to get the details.

Assessing Your Options With FLRT Lite

Off the top of your head, do you know if a POWER7 server will run VIOS 3.1.2.21?

Start by consulting FLRT lite and searching on VIOS. As of this writing, the current VIOS version recommendation is 3.1.2.21.

Then check the VIOS lifecycle information. Version 2.2 has reached end of support, while the projected end of support for 3.1.2.x is November 2023.

I mention POWER7 specifically because a client with that system was wondering about their options. If you go to FLRT and plug in 8202-E4C for the machine type model, you’ll see that 2.2.6.5 is the latest recommended version of VIOS code. For an 8202-E4D model (POWER7+), the recommendation is 3.1.2.21.

What would you do in this situation? Is there enough of a difference between POWER7 and POWER7+ that you would hesitate to deviate from the recommendations? In this case, even when told by support to stand pat, the client was brave and chose to install 3.1.2.21 on their POWER7 box, because that version of VIOS is still supported. It worked. Of course every environment is unique, so your mileage may vary. And ultimately, getting to POWER9 should be your goal if it is at all possible. But I can report that at least one client is now successfully running 3.1.2.21 on POWER7.

VIOS Admin Tips 

I want to again plug the Power Virtual User Group (VUG) technical webinar series. Last month I cited Nigel Griffiths’s first look at the POWER9 HMC, but there’s another excellent presentation from back in May that I neglected to mention. It’s Jaqui Lynch’s deep dive on VIOS: 

Session 106: VIOS—Top Tips for Successful Administration
Speaker: Jaqui Lynch, Information Architect, AIX & Linux Performance Consultant, IBM Power Champion

VIO servers are the most critical part of your system setup. If they are not happy, then no client LPAR is happy. This session provides tips on setting up and maintaining VIO servers including upgrades and patching. Backup and recovery are also covered.

Be sure to download the slides and listen to the replay. As always, she has some great information. Her recent TechChannel article on this topic is also worth your time.

Edit: I still love going to the Grand Canyon

Originally published by TechChannel July 15, 2021

Rob McNelly on automation benefits, AIX patch management with Red Hat Ansible and more

I made one of the classic blunders during a recent trip to the Grand Canyon. This had nothing to do with a land war, or a Sicilian, but as an Arizona resident and avid hiker I should know better: always, always carry your water bottle, especially during the summer.

I venture to the Grand Canyon fairly often. This year alone I’ve been to the South Rim twice, in April and again in June. I’ve also traveled to the North Rim. I’ve hiked and/or biked while camping at Mather, Desert View and the North Rim campgrounds. Most recently, I was driving along the North Rim, making occasional stops at various scenic views and overlooks. Even though it wasn’t anything strenuous, as always, I packed my vehicle with salty snacks and plenty of water. Both are essential when engaging in physical activity in the desert.

However, at one stop, I left my water bottles—along with two 7-gallon containers of water and a cooler full of snacks—behind in my vehicle. I figured I wasn’t going far and that I’d quickly return. I figured wrong. The journey from the car to the overlook was longer than I realized, and the overlook itself branched out into multiple trails. Temps were only in the mid 90s, but with the elevation, low humidity, and lack of shade, the onset of thirst arrives pretty quickly. You can go from feeling fine to having a problem in the blink of an eye.

Of course my provisions did me no good sitting in the car. If I wanted to head out on the trails, I realized I had to first return to my vehicle. Once I’d hydrated and snacked, I loaded my day pack with more food and water. Now I was ready to go back to the overlook and check out the trails for a bit.

Systems Management Planning

In a similar vein, I believe it’s easy to underestimate what we’re getting ourselves into when managing our systems. On first glance we may think we can make a quick fix, but how often do seemingly minor issues end up being something more complicated? Or we may write a few simple scripts and then see that remedy balloon into a hodgepodge of methods and tools that differ from site to site, and sometimes even from admin to admin. Only then do we realize that what works across a handful of systems won’t cut it in a large environment of many systems.

Is there a way out of these dilemmas? Talor Holloway attempts to answer this question in his recent blog post, “AIX Patch Management with Ansible.”

He states: “Leading enterprises today use the Red Hat Ansible automation platform to provision, configure, manage, secure and orchestrate hybrid IT environments. A common misconception is that Ansible is just used to manage the Linux operating system. This is a false belief. Ansible supports Linux, Windows, AIX, IBM i and IBM z/OS environments. This blog will help AIX system administrators get started with Ansible on AIX, and introduce a patching use case … As enterprises move to a modern, enterprise-wide automation strategy with the Ansible automation platform, extending automation to AIX is a great method to simplify and develop consistency in the way AIX systems are supported, all while using the same automation tools that can be used across the enterprise.”

The Benefits of Automation

Automation enables us to do more with less effort, while greatly reducing the potential for human error. It allows us to standardize, allowing for uniformity of tens or hundreds of LPARs in a single environment.

Of course we still must do the work. Planning and testing, which have always been essential to systems management, are also critical with automated solutions. Make a mistake while implementing these tools and you could blow up several of your machines at once. Ultimately though, it’s worth the effort. The more we can automate, the easier our jobs will become. Automation saves time, time that we can spend more productively. In the big picture, we can redirect our newfound time into keeping our skills sharp and passing on our knowledge on to those who will eventually take our place on the raised floor.

VIOS Post-Migration Performance Tip, and Details on the POWER9-Based HMC CR2

On Twitter, Chris Gibson linked to an IBM support document that explains why the emfc_kpr process consumes so much CPU in AIX 7.2 following an upgrade to VIOS 3.1.

The emfc_kpr process is a kernel process that was added for handling the 16Gb and higher speed fibre channel adapters.

Instead of processing the threads via the protocol driver (as it works in previous releases), emfc_kpr process is now processing them.

If there are virtual FC adapters configured in the environment (with VFC), then emfc_kpr works in conjunction with the npivk process.

The emfc_kpr process (and npivk process in NPIV / VFC environment) processes will consume CPU as I/O is moved/passed through.

The doc goes on to explain that the emfc_kpr is designed to enhance the overall adapter driver performance, so despite its CPU usage, overall system performance should improve, though if high I/O throughput is in place, additional CPU resources may be needed.

In a recent Power VUG Technical Webinar Series presentation, Nigel Griffiths takes a “first look” at the new POWER9-based HMC CR2. 

The Advanced Technology team were very impressed with the construction and design of the new POWER9-based HMC (7063-CR2). It operates very much like the older POWER8-based model (7063-CR1), but is faster. It is very much worth the upgrade if you still have the older & slower x86-based HMC. Make sure you are ready for your Power10 servers expected later in 2021, by having the new HMC up and running.

Watch the video replay and download the slides from the July 14 presentation here.

Edit: A nice grab bag of topics

Originally published by TechChannel June 15, 2021

The Power Implementation Quality Standard for Commercial Workloads document has been updated. Get the details from Rob McNelly here.

The Power Implementation Quality Standard for Commercial Workloads document has been updated. Download Version 2.4 (June 2021) here.

This presentation details the expected best practices implementation guidelines for Power Systems hardware, including virtualized systems. As you go through it, notice that, starting with page 9, there’s typically an indicator (in red and in parenthesis) of the most recent update for each slide. The pages that reference the current 2.4 release—either (Mod 2.4) or (Upd 2.4)—have new information, but be sure to review the whole thing. This is one of my go-to reference guides, and I always find useful reminders each time I go through it. Thanks again to IBM senior architect Fredrik Lundholm for his tireless work on this project.

Dandified yum Now Part of the AIX Toolbox

If you work with open source, you’re undoubtedly familiar with yum. (It’s saved many a developer from “rpm hell.”) But it’s time to get on board with dandified yum (dnf), which is now included with the AIX Toolbox.

The need for dnf is easily explained: It supports the current Python 3-based package manager. The open-source community no longer supports the Python 2 package managers upon which yum is based.

One way to set up and run dnf on AIX is with the dnf_aixtoolbox.sh script. (It’s similar to yum.sh, the script for setting up yum.) dnf works on AIX 7.1 TL3 and higher versions. OpenSSL 1.0.2.2001 or higher must also be present on the system. Here’s the script:

            # ./dnf_aixtoolbox.sh -?
            Usage: dnf_aixtoolbox.sh <-d> <-y> <-n> -?
                   -d    Install and setup dnf if yum is not installed.
                  -y    Update yum3 to yum4(softlink to dnf).
                     Use this option if want to redirect all yum commands to dnf.
                      Existing yum-3.4.3 will be updated to yum-4.x.x(dnf)
                  -n    Install dnf where both yum and dnf can coexist if yum is installed already.
                      This is not a recommended option.

            -d option:
                 This option can be used if there is no yum is installed on the machine.
                 This will setup dnf as the default package manger.

            -y option:
                 This is the most recommended option if yum is already installed.
                 This option can be used in two situations.

            If yum is already installed and wants to update yum3 to yum4(dnf).
                      All yum commands are dnf commands as yum will be just a soft link to dnf.

            If yum3(3.4.3) isn’t installed but still like to use yum command which are actually dnf
            commands.

At the end of the document you’ll find examples that show the expected output from running each of the various options.

A New Way to Manage Software and Hardware Service Cases 

IBM is implementing a new web portal for placing hardware service calls. The portal, which will be activated on Saturday, June 19 at 23:59 p.m., replaces the Service Request portal. The newest portal will operate similarly to the recently launched software service portal. As of June 19, all tickets will have a Salesforce Ticket Number (TSxxxxxxx). That means no more PMRs/PMHs for hardware tickets.

To learn more, check out this short video about how to open and manage hardware cases. IBM Support also has FAQs, details about the enhanced work order and Call Home status capabilities and an introduction to Call Home Connect Cloud. 

The new portals are designed to allow customers to manage their hardware and software cases from a single interface. There will be no change to cases opened via voice at 800-IBM-SERV, Call Home, B2B or the IBM Service Center for Cisco Products Portal.

‘Mini-NIM’ Requirements and Other Tidbits

• On Twitter, Chris Gibson noted that IBM has just updated its minimum NIM master levels for VIOS clients. For those of us who manage VIO servers, the version requirements for NIM servers is critical information. Related: “minimum NIM” is kind of a tongue-twister.
• Also on Twitter, Nigel Griffiths shared this 7-minute video devoted to 35 years of AIX. 
• Speaking of NIM, you can learn much more from Jaqui Lynch’s presentation to the Power Systems VUG. Follow the slides and listen to the video. 
• A new certification, IBM AIX v7 Administrator Specialty, is now available. Visit IBM’s certification program site to register for the exam, or to download a sample exam. 
• I’m among a group of IBM Champions who will discuss the present and future of the AIX platform. HelpSystems, a provider of security and automation software, is hosting a discussion, which is set for Wednesday, June 23. It’s free to register.

Historical Perspective

I recently came across this blog, which chronicles random stories about IBM history.

For example: “… do you know that IBM also once embraced biomimicry? That’s right. In 1974 the B in IBM also stood for BEEHIVE!”

Here’s another: “In the mid 1950s, a CIA spook comes to IBM with a request: “Hey Blue suited dudes, make us a machine that can store millions of images that are searchable by keywords. In other words a data lake of images—just like Google Images, except Google is still 40 years in the future.” 

While there have been four entries since March, prior to that, the blog hadn’t been updated since 2017. So don’t count on frequent updates. But I have been enjoying this collection of stories about IBM from decades gone by.

Edit: I still advocate for giving blood and do so roughly every eight weeks

Originally published by IT Jungle June 7, 2021

Over the past year we’ve all heard about essential workers. Essential workers are present in our industry as well. Those who manage and maintain our power grids and technology infrastructures have always played a vital role. It’s easy to overlook these professionals, but they shouldn’t be taken for granted. In 2020, their mostly unnoticed efforts were particularly important.

Don’t get me wrong; I’m not necessarily putting IT workers on the same level as healthcare workers or first responders. My point is simply that during the pandemic, we were reminded that some aspects of our jobs can only be accomplished by people who are present in power plants or on the raised floor in our computer rooms. In many industries the operations teams continued to perform at their work on site.

We were reminded that we can do much of our jobs remotely, and that’s great. However, the shutdown made apparent the need for greater automation. Instead of having someone on site to do a task, we can let the computer do it instead of a human. I find taking repetitive, manual tasks off my daily to do list to be quite satisfying. Take, for instance, setting up processes to automate the creation of a virtual machine. Who wouldn’t prefer to carve out LPARs automatically at the push of a button? How much joy do you feel when you create a script and add it to the scheduler and take repetitive manual tasks off of your plate?

However, some work will always require human hands and eyes. Consider a new server build. (Admittedly this may not be the best example, since many IBM i shops run on a single physical frame, and often for years at a time. But you’ve likely been through a server upgrade at some point in your career, so stick with me.) While much of this process can be performed remotely, let’s start at the beginning. Once your hardware has been received, how do you remotely rack and stack the system, and connect the power plugs into the PDU? How do you remotely plug in a USB disk or a DVD that contains an OS install image? People are still needed for these tasks. You need someone on the raised floor to install and configure the box, at least until it’s powered up and on the network. You need people to troubleshoot network and fiber cable issues. You need CEs to come onsite and replace failed parts. As much as we talk about lights-out data centers, we’re not there yet. Our machines still require human intervention.

A Home/Office Hybrid

No one wants to relive 2020, but we have learned lessons from it just the same over the past year. For instance, remote work is likely here to stay. In our industry and others, people came to value working from home, and they want the flexibility to continue to do their jobs away from the office on at least a part-time basis. Forrester predicts that most companies in the United States and European will employ a hybrid work model post-pandemic. This is echoed by a Citrix survey indicating that “90 percent of respondents have no interest in returning to office work full time once the COVID-19 pandemic is over. More than half prefer a hybrid working model where they can work from home most or all of the time, while 18 percent want a hybrid model where they work from the office more.” Bloomberg shares anecdotes where employees are quitting instead of giving up working from home.

Author Scott Berkun suggests that CEOs roll with the change in attitude:

Remote work is seen as a threat to many CEOs simply because of their fear of change and resistance to progress. That fear leads to an irrational rejection of remote work, instead of a thoughtful examination of where it has succeeded and what can be learned.

Incidentally, no one had to make the case for working at home to me. I’ve done it for years: For me, quoting myself here, working at home was an easy choice. “I love being able to get going first thing in the morning while I’m fresh and alert. On weekdays at least, I’ll typically awaken thinking of work anyway. So rather than sitting through a commute, worrying about losing time, I can get right to it…. my dedicated office workspace has long been in place. I have fast Internet, a full-size multi-monitor setup, and my cherished old school tools: an actual landline and a vintage Model M keyboard.”

Naturally though, my own work routine also underwent a transformation. Even though I was used to working from home, I’d never worked exclusively from home. Prior to 2020 I traveled around the country, working at customer sites and attending conferences and classes (while racking up frequent flier miles and hotel and rental car points along the way). But I spent much of this past year helping business clients scale their infrastructures to allow for higher system utilization. Many of us had to enable our employees to be able to work remotely. Sometimes that involved beefing up bandwidth requirements to allow for more employees to connect remotely via VPN and use collaboration tools.

Another thing I’ve learned is that collaboration has its limits. As I write this, I have Slack open, along with multiple email inboxes. Throughout the day I can expect to join meetings using Webex, GoToMeeting, Zoom, or Teams. I’ll use shared documents, in addition to those I send and receive in email. And I’ll converse with coworkers via Teams, Google Hangouts and Jabber, among other messaging platforms.

Of course, the advantage to messaging – and even text and email, to an extent – is it’s meant to be non-intrusive. You’re allowing the other person to take their time and respond when they’re available. That’s considerate, and consideration is important. But sometimes you just have to cut through all the clutter and talk to people in real time. Talking one on one clears up misunderstandings. Sometimes it’s the simplest way.

Give Yourself A Pat On The Back

Over the last year, I was reminded that time is short. Fortunately, my family’s health wasn’t directly affected by Covid, but I still felt its impact. In the past year one of my friends died from brain cancer while another lost a child to Leukemia. The pandemic made that all the more difficult. Hospital visits were restricted, both in terms of hours and the number of people allowed in the room. That’s assuming visitors were allowed at all; I was turned away more than once. With the limits on gatherings, funeral services were also different.

Certainly, I wasn’t the only one to be reminded of what is truly important. Another example: blood donation. I’d donated sporadically for years, but seeing the gratitude of these families for those who made it possible for their loved ones to get transfusions, plasma and other blood products inspired me to give blood as often as I can, roughly every eight weeks. Please consider giving blood if you are able.

Things are starting to open up. Where I live fewer businesses are requiring masks. Seemingly it’s just a matter of time before I get back on the road for client visits and in-person conferences. While I enjoyed learning about tiger sanctuaries and chess players, I’m more than ready for a return to near normalcy.

How ever you managed to get through the past year, I believe you, the IT professional, are worthy of appreciation. By simply doing your job, by applying your unique skills, you made vital contributions. Your sacrifices went largely unseen. You work weekends. You work holidays. You’re on call at all hours. When systems require patching or operating systems require upgrading, you may get a change window of 1 to 3 a.m. on a Sunday. You have to make that work, and you do.

I’m reminded of the old Dunkin Donuts TV ads. A guy would head out the door at the crack of dawn and say to himself, “It’s time to make the donuts.” The world needs the people who make the donuts. The world also needs techies, and during a tough time, you delivered.

Rob McNelly is a senior Power Systems solutions architect doing pre-sales and post-sales support for Meridian IT, headquartered in Deerfield, Illinois. McNelly was a technical editor for IBM Systems Magazine, and a former administrator within IBM’s Integrated Technology Delivery and Server Operations division. Prior to working for IBM, McNelly was an OS/400 and IBM i operator for many years for multiple companies. McNelly was named an IBM Champion for Power Systems in 2011, an IBM Champion Lifetime Achievement recipient in 2019, and can be reached at rob.mcnelly@gmail.com.

Edit: I should still be better prepared for my rides in the desert

Originally published by TechChannel May 18, 2021

Rob McNelly on why adequate system maintenance can help you prepare (and avoid) system problems

As I found myself walking through the desert, pushing a bicycle with a flat tire, I wondered how I got here.

That’s not a metaphor; that’s life in Arizona. With the cactus and tumbleweeds and other assorted spiky, poky things, it’s a dangerous place for tires and tubes. It’s not a great place for people, either, once the temperatures hit triple digits, as was the case that day.

I probably could have called someone to pick me up, but that would just add insult to injury. Besides, I didn’t have far to walk, and I had enough water on hand. So I trudged home in the heat, wondering how I could have avoided this fate.

For starters, I could have filled my tire tube with slime. That’s a real thing. It seals any holes in tire tubes. I wouldn’t have needed to do anything else.

Or I could have brought along a spare bike tube, pump, and patch kit. Then I could make repairs on the spot. Or I could have simply replaced the tire itself. It was old and the tread was thinning.

Ultimately I realized this was on me. This outcome was entirely foreseeable, and I’d neglected to adequately prepare for it.

Does that sound familiar? Isn’t this often the case with your system maintenance? Once you determine the cause of the problem, it’s glaringly obvious that something was neglected along the way. Say you open a PMR and IBM informs you that your issue is a known defect. Had you patched your system when that SP or TL was first released, months or even years prior, the bug would not have affected you. Or maybe you need a physical frame taken down so a CE can replace a part. Many components are “hot swappable” these days, but not everything. Wouldn’t it have been nice if you’d have prepared for this eventuality by simply keeping a spare frame with free resources available in your environment? That way you could LPM the workload to it, and the necessary work wouldn’t affect anyone at all.

But rather than plan, you hoped for the best. Or perhaps you concluded that if ain’t it broke, don’t fix it. And the outcome was entirely foreseeable—as well as entirely avoidable.

Make the Case for Maintenance 

Back in the 1980s Castrol Motor Oil reminded TV viewers that “if you make things hard on your engine, your engine will make things hard on you.” (While a surprising number of old Castrol ads are archived on YouTube, I couldn’t find that particular one. But this is similar.) Like your car, your system is a valuable and complex piece of machinery that requires care. Fall too far behind on patching and basic maintenance, and the simple becomes much more complicated than it should be. Updates become large upgrade projects. Ignoring maintenance will ultimately leave you running old, unsupported hardware and out of date OSes, with no easy path forward. Technical debt will be paid, one way or another. 

What should be done? Start by letting those at the C-level know that patching is important, and potentially a huge cost savings over inaction. Maybe use the car analogy in a gentle reminder. If you can see the need to change your oil or fill your tank (or charge your battery in the case of electric vehicles), you should be able to see the need for system maintenance. You know you can’t ignore that oil change reminder sticker in your windshield. Sure you can put it off, but not forever. The same applies to your machines.

We may need to convince others of the importance of maintenance, but we admins should know from bitter experience why it matters. It’s a sinking feeling when you realize how easy it would have been to ensure your backups would function before the need arose to actually restore your machine. What about that mksysb you’re taking? Have you audited it to make sure the images are not only being created, but are actually usable? Sure, by writing a script you theoretically can set it and forget it in cron, but neglect to check on the results of that backup script, and you’ll end up with Schrodinger’s backup:

“The condition of any backup is unknown until a restore is attempted.”

Certainly your VIO mappings and VLANs and other configurations are saved somewhere, right? And your VIO servers are backed up as well, so they can be easily recreated if needed? What about your HMC? Is it easily recoverable? Are the configurations backed up? Did you run and keep hmcscanner reports so you know how everything was set up in your environment? Do you have the information so that shared Ethernet adapters and etherchannel devices can be recreated if needed? Have you actually done this? What about mapping your NPIV and vSCSI disks? And do you know which disk drivers you need to load?

There’s even more to consider, but you get the point. It all boils down to being prepared for the unexpected. Any and all of these problems can leave you on a slow, humbling walking through the desert, asking yourself why you weren’t better prepared and hoping you have enough water to see you through.

And that is a metaphor.

Edit: This was published as an ebook by TechChannel

Rob McNelly explains what questions you and your team should be
asking to make sure data is locked down

Click to access TechChannel_ebook_SMB-Security_JUNE2021.pdf

I will also archive it here in case the original link stops working in the future.

You’ve seen the headlines about malware attacks and cyberhacks. Whether it’s a competitor looking to steal your secrets or criminals looking to extort money, system administrators have myriad reasons to be wary. After all, the only absolutely secure system is one that is powered off.

Luckily, if you’re running AIX on IBM Power Systems hardware, you are officially “secure” and don’t need to take further action (in case you couldn’t tell, that’s sarcasm).

Don’t get me wrong; AIX is great. It’s my favorite OS. But it still requires monitoring and patching, and that’s for starters. If you don’t believe me, check IBM’s APAR security information or CVE vulnerability data.

It may be true that Windows and Linux systems, which number in the millions, are higher profile and thus more commonly targeted. However, that’s no reason for AIX admins to be complacent. If anything, systems running AIX make more tempting targets for bad actors. Look at it this way: The data held on AIX systems is incredibly valuable.

These machines typically run mission-critical workloads and essential databases and applications for some of the world’s largest enterprises. What are the ramifications of someone gaining access to or corrupting this data? What happens if records are deleted or destroyed? Yes, most AIX systems are behind a firewall, and most large corporate environments have disaster recovery sites and detailed recovery plans. Again though, that’s not enough. More must be done to reduce the chances of a damaging attack.

Put Yourself on Notice: IBM Support

I like to keep up to date on the latest known vulnerabilities by subscribing to IBM’s notifications. You can bookmark the links I cited earlier, or just do what I do and register for IBM updates. I receive weekly emails from IBM. Go to the IBM Support site to subscribe and manage your subscriptions and delivery preferences.

While I prefer weekly updates, you can opt for daily email. You can also limit update topics to ensure the information you receive is relevant.

Once you check these boxes, ask yourself some questions about your own environment. For instance, if an attacker gains access to your internal network, how quickly or easily could you identify the vulnerability? Are unnecessary services running on your machine? It’s harder to attack a system that’s listening on only a limited number of ports.

I mentioned firewalls: They’re a nice line of defense, but attackers can still beat them and gain access via the network. They could gain a foothold by compromising VPN credentials or some Windows or Linux machine on the network, and then move laterally within your organization by behaving as an authorized network user. Your network team should be watchful for unusual behavior such as logins at odd hours or atypical actions.

Asking the Tough Security Questions

To see if you are covered, ask yourself these 18 question

1. Do the user IDs on your system have strong passwords?

2. Have you changed your default password algorithm?

3. Have you disabled or deleted accounts that are no longer needed?

4. Are you authenticating via LDAP or some other central service, or are you trying to manually manage user IDs across your machines?

5. Once users log in, are they allowed to escalate their privileges via sudo or some other mechanism?

6. Are those permissions regularly audited?

7. Are the sudo logs themselves audited?

8. Are you tracking attempts, successful or not, to log into your system? Put a machine on port 22 on the public facing internet and see how quickly it gets inundated. If you’re seeing that sort of activity behind your firewall,
something may not be right.

9. If you’re tracking logins, are log files being monitored and reviewed?

10. Do you have a security information and event management (SIEM) server that actively checks logs across your environment?

11. Are log files growing without being rotated, or are they allowed to grow indefinitely? Considering the huge amount of disk that we can allocate to filesystems these days, log file size may seem insignificant, but rotating these logs is still a good idea.

12. Do you keep logs locally or send the files to a central system? This information can help diagnose an intrusion, particularly if an attacker gains access to a machine and alters the files stored there. Of course, if an attacker accesses the logging machines and deletes those files, that’s another matter.

13. Are your systems regularly patched? Besides the OS, are you up to date on patching system firmware, device firmware and any VIO servers that are in use?

14. For those who continue to rely on legacy applications and older AIX versions, are you taking extra precautions? Those using unsupported hardware and software don’t have the options of opening a problem ticket with IBM support or applying security patches. If you’re dealing with these limitations, you must be extra vigilant in assessing and monitoring risks to your environment.

15. What are your procedures, who gets notified, and what actions are taken when an intrusion attempt is detected or recognized after the fact?

16. Who determines when systems should be removed from the network, and who decides how to analyze the system after an event occurs?

17. At what point do you declare a disaster and move operations to another location?

18. Do you have a disaster recovery plan?

Help From the Outside: Lab Services and Documentation

IBM Lab Services for Power Systems or your IBM Business Partner can help you assess your organization’s security and compliance practices and procedures.

Another option is to engage a penetration testing company. Penetration testers simulate attacks to determine how your system would hold up against the real thing, and how well your staff responds to notifications of anomalies in real time. Knowing that there was no detection of an attack is valuable information as well.

While this overview offers a few things to keep in mind as far as managing the security of your systems, it is by no means intended to be an exhaustive list. Rather it is meant to help jump start conversations in your organization to start considering how important your data is and what you can do it keep it safe. I encourage you to read this detailed look at AIX security strategies authored by lifetime IBM Power Champion Jaqui Lynch

Edit: Are you current?

Originally published by TechChannel April 22, 2021

Rob McNelly on why AIX has stood the test of time, the new POWER9-based HMC and how IBM certifications have evolved

I believe AIX has stood the test of time. This opinion is based in part on the fact that legacy AIX documentation is still relevant. For example, I recently downloaded two Redbooks, “IBM Certification Study Guide eServer p5 and pSeries Administration and Support for AIX 5L Version 5.3” and “IBM eServer Certification Study Guide – pSeries AIX System Support.” Seeing “eServer” in the titles gives you an idea of how far back I went, but for the record, that admin and support doc was published in 2006, and last updated in 2010. The certification study guide came out at the end of 2001, and was updated in 2004.

Honestly, a lot of this information is still useful. Of course not everything translates to today’s highly virtualized, flash storage-based environments, but so much does. This speaks to the design and the thought that’s been put into the OS from the start. The commands and concepts have always been well thought out. That the system can be tuned to fit unique workloads is very impressive. And the value of IBM Support, which quickly diagnoses and helps resolve problems, is something I still take for granted.

Did you need to relearn everything when you went from AIX 6.1 to 7.1, or 5.3 to 6.1, or 4.3.3 to 5.1? No. Do the smitty menus and fastpaths look the same? Do your old scripts typically continue to run when you upgrade to a new version of the OS? Do you have LPARs that have actually migrated between multiple versions of the OS? And (as much as I hate to even ask), are those LPARs that are running unsupported versions still going strong? That would be yes, yes, yes and yes.

As we consider our next moves—whether it’s patching and updating the OS or planning for cloud migration, the move to POWER10-based servers or an upgrade to AIX 7.3—we can be secure in knowing that the framework will remain the same.

Checking Your Current AIX/VIOS Versions

Nigel Griffiths has a great analogy about the AIX/VIOS versions you should be running: 

“You always get your cars brakes, shocks, tires & lights checked + updated every year, as you value your family + friends. The same goes for operating systems. Failing to update to current supported versions = you lose your job! Here is my take on the AIX/VIOS we all should be using.”

Use FLRT LITE to check for the current AIX and VIOS recommendations.

A New POWER9-Based HMC 

In case you missed it, IBM announced a newer HMC model based on the POWER9 processor: 

The Power HMC (7063-CR2) is a dedicated rack-mounted workstation that helps you to configure and manage system resources on Power servers using POWER7, POWER8, or POWER9 technology-based processors. The HMC connects to one or more managed systems to perform the following primary functions:

• Provide a console for system administrators and service providers to manage server hardware
• Deliver basic virtualization management through support for configuring logical partitions (LPARs) and dynamic resource allocation, including processor and memory settings
• Detect, report, and store changes in hardware conditions
• Act as a service focal point for service providers to determine an appropriate service strategy
• Display operating system session terminals for each partition
• Provide the call home focal point for managed servers
• Display ASMI menus for managed servers

The announcement letter lists these standard hardware attributes:

• 1U base configuration
• POWER9 130W 6c CPU
• 64 GB (4 x 16 GB) or 128 GB (4 x 32 GB) of DDR4 system memory
• 2 x 1.8 TB SAS SFF 2.5-inch hard disk drive (HDD) RAID 1
• Rail bracket option for round-hole rack mounts
• Two USB 3.0 hub ports in the front of the server (option to remove)
• Two USB 3.0 hub ports in the rear of the server
• Redundant 900W power supplies
• 4 x 1 Gb Ethernet ports
• 2 x 10 Gb Ethernet Ports on optional PCI adapter
• 1 x 1 Gb baseboard management controller (BMC) Ethernet port

To see it in action, check out this Nigel Griffiths video. 

Power Systems VUG Covers Recent Announcements, AIX Anniversary 

As noted, AIX turns 35 this year. The latest Power Systems Virtual User Group presentation acknowledges the anniversary and covers recent AIX announcements. View the PDF and listen to the replay.

Among other changes, expect to see new AIX certifications. An AIX Foundations exam is coming soon, and other AIX certifications (including advanced certifications) will follow.

Here’s a post about the evolution of IBM certifications. Look for more granular levels of certification. IBM is calling them foundational skills, intermediate skills, and advanced skills. There are differences and distinctions between them:

• Foundational activities are geared toward learners who are new to the subject matter and seeking to learn basic concepts and build foundational knowledge in support of efforts to gain a working knowledge of the topic. These credentials are typically issued to individuals with little or no prior knowledge and experience with the subject matter represented by the credential.
• Intermediate level activities are geared toward learners who have acquired some degree of competence in the covered topic resulting from prior training, education and/or work experience. Intermediate level activities are for learners who seek to build upon foundational knowledge, refine and better hone their skills, and advance their understanding of the topic.
• Advanced activities are tailored toward learners who have already achieved a higher degree of technical competence in the subject matter resulting from expanded training and supplemental work experience. Advanced level activities are for learners who wish to build upon intermediate knowledge and field experience toward the achievement of mastery in a specific technical area.

Other notes from this post:

• Thought Leader credentials are earned by subject matter experts who are frequently consulted for their mastery level knowledge and skills. They are able to apply those skills in the most challenging situations and are highly effective at coaching and mentoring others in doing the same. A Thought Leader is an innovator and respected authority on the related subject matter. Thought Leader skill demonstration is typically validated by SME, board review, evidence submissions, and verified experiential activities.
• IBM Certificate-Validation of knowledge, skills and abilities achieved through the aggregation of multiple learning experiences structured as a prescriptive learning pathway supporting skill development within a more comprehensive area of study.
• IBM Professional Certification provides validation of IBM technology-oriented knowledge, skills and abilities for essential job roles through administration of proctored examination and/or performance-based testing. The certification assessment is independent of any specific educational event or related learning opportunities. Certification is also intended to measure or enhance continued competence through re-certification or renewal requirements. Re-certification may require mandatory continuing education hours, proctored reassessment, non-proctored reassessment or a combination thereof.

Disclosure: I did some volunteer work on the foundational test. Part of our discussion involved looking at past study guides.

Edit: Still one of my favorite trails, I returned in December and made it to Plateau Point.

Originally published by IT Jungle April 26, 2021

In my last piece, we got Yum and Bash running on IBM i. Now let’s build on that foundation to find an open campsite.

I’m serious. Recently I used an IBM i LPAR to schedule a hiking and camping trip to the Grand Canyon. Where to begin? My bio, perhaps. Here it’s mentioned that “Rob enjoys camping, hiking, biking, and backpacking through the mountains of Arizona. . . . His favorite trip was hiking to the blue waters of Havasupai, and he is planning on hiking the Grand Canyon in the near future.”

So last summer I went camping and ended up riding my bike around the South Rim. I told a friend about my plans to do more exploring there this year, including a rim to rim hike. Basically, he called me out. He wanted to do the hike, but he was also sick of hearing me go on about it.

Attempting a rim to rim hike in a single day isn’t recommended. It’s best to spend a night or two resting at the bottom at Phantom Ranch, one of the developed campsites, or to find your own spot in the back country (assuming you can obtain the proper permits and reservations). Of course the appeal of day hikes is that no permits are needed. However, you must be properly conditioned for this challenge:

“Over 250 people are rescued from the canyon each year. The difference between a great adventure in Grand Canyon and a trip to the hospital (or worse) is up to YOU. DO NOT attempt to hike from the rim to the river and back in one day, especially during the months of May to September.”

Though it was early April and not that hot yet, I took this advice. Rather than go rim to rim, I thought we’d try something of a practice hike down to Indian Garden. That covers around 10 miles round trip with the accompanying elevation changes. At that point we could gauge our comfort level, and if we felt good we could continue the hike to Plateau Point, which would add about three miles before we’d turn around. For some context, depending on the route chosen, if you do a rim to rim hike starting at the North Rim you’ll descend 6,000 feet, and then come up 4,500 feet to the top of the South Rim while covering around 24 miles.

Although plenty of free camping is available on National Forest land just south of the park entrance, I wanted to sleep closer to the trailhead, so I tried to get a spot at the developed Mather Campground near the South Rim of the Grand Canyon. The recreation.gov website had no availability at Mather for the nights I wanted, even though I’d periodically check and refresh. Eventually I got to wondering about other options, and sure enough, a site called campsitephotos.com has a campsite assist offering. This seemed to be what I was looking for.

I clicked on the arrow and entered my destination:

There were options to narrow the search, but I wasn’t picky.

I chose a Matrix scan:

Then I entered some potential dates:

I provided my cell number to receive text alerts:

Then it hit me up with the pricing options:

That was affordable enough, but what if I wanted to make changes? What if I wanted to check more often than every 15 minutes? How hard was it to roll my own website scraper? I searched further and found some interesting code on GitHub. (More readable versions are typed below each of the following screen shots, which came from: https://github.com/banool/recreation-gov-campsite-checker)

Campsite Availability Scraping

This has been updated to work with the new recreation.gov site and API!!! This script scrapes the https://recreation.gov website for campsite availabilities.

Note: Please don’t abuse this script. Most folks out there don’t know how to run scrapers against websites, so you’re at an unfair advantage by using this.

Example Usage
$ python camping.py --start-date 2018-07-20 --end-date 2018-07-23 --parks 232448 232450 232447 232770
TUOLUMNE MEADOWS: 0 site(s) available out of 148 site(s)
LOWER PINES: 11 site(s) available out of 73 site(s)
UPPER PINES: 0 site(s) available out of 235 site(s)
BASIN MONTANA CAMPGROUND: 0 site(s) available out of 30 site(s)

---

Installation

I wrote this in Python 3.7 but I’ve tested it as working with 3.5 and 3.6 also.

python3 -m venv myvenv
source myvenv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt
# You're good to go!

---

For all I know this code may be in use or even modified by Campsite Assist. In any event it seemed like a great tool to run on an IBM i LPAR. And sure, many people may see Linux as a more natural environment for something like this, but there’s value in getting these tools to run on IBM i, particularly when it’s this simple.

I logged in via putty and checked to see if Python3 was installed:

	bash-4.4$ python3
	bash: python3: command not found

I used the find command to check for yum:

	bash-4.4$ find / -name yum -print
	/QOpenSys/pkgs/bin/yum

Then I used Yum to install Python3. I omitted many of the messages and skipped to the end:

	bash-4.4$ /QOpenSys/pkgs/bin/yum install python3
	Installed:
	  python3.ppc64 0:3.6.12-1
	Dependency Installed:
	  libreadline8.ppc64 0:8.0-1

Complete!

I also wanted to install git so I could copy the code from GitHub.

	bash-4.4$ /QOpenSys/pkgs/bin/yum install git
	Installed:
	  git.ppc64 0:2.26.2-3
	Dependency Installed:
	  libpcre2-8-0.ppc64 0:10.35-1

I wasn’t sure where git was installed, so I ran find and then used git to clone the website:

	bash-4.4$ find / -name git -print
	/QopenSys/pkgs/bin/git
	bash-4.4$ /QOpenSys/pkgs/bin/git clone https://github.com/banool/recreation-gov-campsite-checker.git 
	Cloning into 'recreation-gov-campsite-checker'...
	remote: Enumerating objects: 16, done.
	remote: Counting objects: 100% (16/16), done.
	remote: Compressing objects: 100% (12/12), done.
	remote: Total 161 (delta 7), reused 11 (delta 4), pack-reused 145
	Receiving objects: 100% (161/161), 42.65 KiB | 2.24 MiB/s, done.
	Resolving deltas: 100% (77/77), done.
	bash-4.4$ ls -ld rec*
	drwxr-sr-x    4 rob 0             12288 Apr  3 07:20 recreation-gov-campsite-checker
	bash-4.4$ cd recreation-gov-campsite-checker/
	bash-4.4$ ls
	README.md                      camping.py                     fake_twitter_credentials.json  notifier.py                    	other                          requirements.txt               setup.py

Now it was just a matter of running through the installation instructions listed at the beginning of this section:

	bash-4.4$ /QOpenSys/pkgs/bin/python3 -m venv myvenv
	bash-4.4$ source myvenv/bin/activate
	(myvenv) bash-4.4$
	(myvenv) bash-4.4$ pip install --upgrade pip
	Collecting pip
	  Downloading 	https://files.pythonhosted.org/packages/fe/ef/60d7ba03b5c442309ef42e7d69959f73aacccd0d86008362a681c4698e83/pip-21.0.1-py3-none-any.whl (1.5MB)
	    100% |################################| 1.5MB 3.1MB/s
	Installing collected packages: pip
	  Found existing installation: pip 18.1
	    Uninstalling pip-18.1:
	      Successfully uninstalled pip-18.1
Successfully installed pip-21.0.1
	(myvenv) bash-4.4$ pip install -r requirements.txt

There was quite a bit of output once I’d finished:

Successfully installed Click-7.0 appdirs-1.4.3 attrs-18.2.0 black-18.9b0 certifi-2018.11.29 chardet-3.0.4 fake-useragent-0.1.11 future-0.17.1 idna-2.8 isort-4.3.4 oauthlib-3.0.1 python-dateutil-2.8.1 python-twitter-3.5 requests-2.21.0 requests-oauthlib-1.2.0 six-1.15.0 soupsieve-1.8 toml-0.10.0 urllib3-1.24.2

I got some errors when I ran the code, which uses emojis to alert you on the status of the search. Rather than try and make that work, I changed the emojis to English text by running:

	vi camping.py 

Then I edited 29 and 30:

	  +29 SUCCESS_EMOJI = "yay"
	   +30  FAILURE_EMOJI = "boo"

At this point it ran as expected. I tried a few different date ranges to see what was being returned:

(myvenv) bash-4.4$ python3 camping.py --start-date 2021-04-06 --end-date 2021-04-08 --nights 2 --parks 232490
There are no campsites available :(
boo MATHER CAMPGROUND (232490): 0 site(s) available out of 357 site(s)
(myvenv) bash-4.4$ python3 camping.py --start-date 2021-04-06 --end-date 2021-05-08 --nights 2 --parks 232490
There are campsites available from 2021-04-06 to 2021-05-08!!!
yay MATHER CAMPGROUND (232490): 23 site(s) available out of 357 site(s)

To view the raw data being returned, turn on debug mode with — debug. You’ll then get specific campsite IDs and available dates:

(myvenv) bash-4.4$ python3 camping.py --start-date 2021-04-01 --end-date 2021-04-09 --parks 232490 
--debug   2> test.out
	(myvenv) bash-4.4$ cat test.out 
	2021-04-03 15:38:36,066 - 19668 - DEBUG - Querying for 232490 with these params:
	 {'start_date': '2021-04-01T00:00:00.000Z'}
	2021-04-03 15:38:39,031 - 19668 - DEBUG - Information for park 232490: {
	  "4079": [],
	  "4080": [],
	  "4081": [],
	  "4082": [],
	  "4083": [],
	  "4084": [],
	  "4085": [
	    "2021-04-28T00:00:00Z"
	  ],
	  "4086": [],
	  "4087": [],
	  "4088": [],
	  "4089": [],
	  "4090": [
	    "2021-04-27T00:00:00Z"
	  ],
	  "4091": [],
	  "4092": [
	    "2021-04-28T00:00:00Z"

That’s just a taste. The actual file has tons of additional information. At this point, I created a quick and dirty script called test.bash:

	python3 camping.py --start-date 2021-04-01 --end-date 2021-04-09 --parks 232490 
	--debug   2> test.out
	cat test.out | grep '2021-04-0[5678]T'

The first line ran the code and sent the debug output to a file; the second line read through the file and searched for a date string. I broadened my search to April 5-8 rather than April 6-7 so I could see results even when my target dates were unavailable. Wild cards and regular expressions are beyond the scope of this article, and I realize there are probably more elegant ways to accomplish this, but I’d be happy to hear how you would have done it.

I made the script executable by running:

	(myenv) bash-4.4$ chmod u+x test.bash

Then I ran it with:

	(myvenv) bash-4.4$ ./test.bash
	There are no campsites available :(
	boo MATHER CAMPGROUND (232490): 0 site(s) available out of 357 site(s)
	    "2021-04-05T00:00:00Z",
	    "2021-04-07T00:00:00Z",
	    "2021-04-07T00:00:00Z"
	    "2021-04-08T00:00:00Z"

At this point I created a simple loop that would run on the command line.

	(myvenv) bash-4.4$ while true
	> do date
	> ./test.bash
	> sleep 300
	> done
	Sat Apr  3 08:34:35 CST 2021
	There are no campsites available :(
	boo MATHER CAMPGROUND (232490): 0 site(s) available out of 357 site(s)
	    "2021-04-05T00:00:00Z",
	    "2021-04-07T00:00:00Z",
	    "2021-04-07T00:00:00Z"
	    "2021-04-08T00:00:00Z"

Rather than hammer the site, I gave it a 5-minute delay (300 seconds). Yes, even 5 minutes may be excessive, but I figured it wouldn’t need to run for long. In any event, that interval worked for me. After about 25 minutes, an open campsite that fit my criteria became available.

During this test run you can see the changes in availability:

	Sat Apr  3 08:34:35 CST 2021
	There are no campsites available :(
	boo MATHER CAMPGROUND (232490): 0 site(s) available out of 357 site(s)
	    "2021-04-05T00:00:00Z",
	    "2021-04-07T00:00:00Z",
	    "2021-04-07T00:00:00Z"
	    "2021-04-08T00:00:00Z"
	Sat Apr  3 08:39:38 CST 2021
	There are no campsites available :(
	boo MATHER CAMPGROUND (232490): 0 site(s) available out of 357 site(s)
	    "2021-04-05T00:00:00Z",
	    "2021-04-07T00:00:00Z",
	    "2021-04-08T00:00:00Z"
	Sat Apr  3 08:44:39 CST 2021
	There are no campsites available :(
	boo MATHER CAMPGROUND (232490): 0 site(s) available out of 357 site(s)
	    "2021-04-07T00:00:00Z",
	    "2021-04-08T00:00:00Z"
	Sat Apr  3 08:49:42 CST 2021
	There are no campsites available :(
	boo MATHER CAMPGROUND (232490): 0 site(s) available out of 357 site(s)
	    "2021-04-08T00:00:00Z"
	Sat Apr  3 08:54:45 CST 2021
	There are no campsites available :(
	boo MATHER CAMPGROUND (232490): 0 site(s) available out of 357 site(s)
	    "2021-04-08T00:00:00Z"

To get this going, you need to know your campground ID. Search for your campground on recreation.gov; the ID is in your title bar.

This will work for any campsite. Emojis aside, this code worked without any modifications. Everything ran as described on GitHub, and it was running on a Power Systems server. Incidentally, I also tried it on a Linux machine to make sure there were no differences between the platforms. Indeed, everything was the same, right down to the emoji issue.

As it happened, my buddy couldn’t make that trip, so my son joined me for some camping. We hiked down to Indian Garden and back along the Bright Angel trail. We got a late start, so we made sure to pack it in before dark rather than continue on to Plateau Point. Physically I felt up to it, but I also realized I’ll need to train more for the real deal; coming back uphill is no joke. My legs were sore and stiff for days after.

Knowing that I had a reservation at a campsite that was relatively close to where we started/ended the hike made the experience even more enjoyable. Sure, this isn’t exactly a traditional use for IBM i, but my tale confirms what you should already know. Using modern tools and techniques, this OS is capable of solving all kinds of interesting problems.

Edit: Have you started your upgrades yet?

Originally published by TechChannel March 10, 2021

Rob McNelly on AIX and Power hardware improvements, and where the AIX roadmap might lead beyond 2023

AIX turns 35 this year. While it’s fun to look back (here’s the IBM announcement letter from 1986), there’s also ample reason to look ahead, especially since IBM recently stated that it intends to release AIX 7.3 before the end of this year.

Previous AIX releases have included an open beta program, and it’s my understanding that this will be the case with 7.3 as well. Expect to see additional open-source packages bundled and supported with the base OS out of the box. The new release will run on POWER8 processors and later, so it may be time to consider migrating from any POWER7 or older hardware that’s still running in your environment.

I love AIX, to the point that I almost feel protective of it. I’ve said for years—most recently in 2018—that AIX isn’t going anywhere. Still, some pushback persists, so I feel like I need to continue to remind everyone that this OS runs critical workloads throughout the business world. For instance, I’ll get asked, “When is the next release coming?” even though people should understand that new capabilities are being brought to AIX on an ongoing basis through the application of service packs (SPs) and the release of new technology levels (TLs).

It’s a matter of aesthetics, but had IBM opted to call this release 8.1, that stamp of newness would have provided doubters with additional reassurance. Of course there’s a flip side, too. The leaps from AIX 5.3 to 6.1 and AIX 6.1 to 7.1 made some clients and ISVs skittish. From this perspective, the steps from 7.1 to 7.2 to 7.3 seem less urgent.

In any event, there is intrinsic value in knowing that a new release is on the way. Consider the tone of this piece from The Register: “While IBM has promised years more support for AIX, news of an update will be welcome as a sign of ongoing commitment and because the last major drop of the OS 2015’s version 7.2.”

However you’re looking at this, keep in mind that IBM’s plans for AIX go far beyond 2021. The AIX lifecycle currently extends to 2023. For some though, a two-year window isn’t much reassurance. “What will happen in 2024?” I hear. To that, I encourage people to check out this AIX Executive Strategy paper. This document includes a roadmap that takes AIX beyond 2030. You’ll need an IBM ID to download it, but it’s worth your while. You’ll find nuggets throughout that illustrate the value of our favorite OS. For instance:

“AIX is deployed across a variety of industries such as finance, manufacturing, retail, telecommunications, healthcare, travel and government, along with many others. … As IT infrastructure expands into new workloads, the ability of Power Systems and PowerVM virtualization allows AIX, IBM i and Linux to run side by side for efficient consolidation and optimization of data exchange and processing between these different environments. Power Systems is unique in its capabilities to host this wide range of solutions efficiently”

And, as has been the case for years, IBM provides “binary compatibility guarantees to ensure that clients can run their workloads on new AIX releases such as 7.3, on the latest Power platform without having to worry about recompiling or rewriting applications.”

So, to sum up: The capabilities of AIX are continually being improved. Power hardware is capable of running multiple operating systems on the same frame, and, as always, AIX is running mission critical applications in multiple industries around the world. Does that sound like a dying OS and ecosystem to you?

There’s other recent IBM news of interest to AIX users. Brandon Pederson notes that IBM is encouraging clients to upgrade to POWER9 now by allowing them to lock in a price for POWER10:

“After announcing the next generation POWER processor, we often hear from clients, “oh, I’ll just wait to upgrade my infrastructure until then!” But why wait? There are immediate performance, availability and security benefits to be had by upgrading to POWER9 now. So, to help put clients on the path to POWER10 but also take advantage of POWER9 right away, we are announcing a special offering for our scale-out servers. Available starting today, the IBM Power Systems Flexible Trade Up Offer for the S922 and S924 will help clients upgrade now to POWER9 and then to POWER10 when available at a predictable price.”

Anyway, now is a good time to reflect on all the reasons you love AIX. To that end, Nigel Griffiths has a link to some of the AIX “Best Bits,” where he highlights key parts of the OS and ecosystem that are often taken for granted. And Prenessa Lowery tells us what to expect as IBM officially marks the 35-year anniversary:

“You will have the opportunity to hear from IBM executives, SMEs, AIX influencers, AIX customers and more as they share their AIX story! Also, you will get a chance to gain more insights on how AIX will prepare clients for the future.”

So there’s a whole lot going on with AIX, now and well into the future. As we take the time to celebrate where we came from, we should also be excited for what’s ahead. Hopefully the knowledge that AIX 7.3 and POWER10 are on the horizon are compelling reasons to help convince you that your favorite operating system is alive and well and worth your continued attention.

Edit: It might be time to start looking at dnf

Originally published by IT Jungle March 15, 2021

In the March 1 edition of The Four Hundred, I noted that an emphasis on things like system/application modernization and open source solutions gives IBM i newcomers a degree of comfort with the platform. I also made the point that no matter how long you or I have been at this, there are always people, young or not so young, who are new to the platform and come to this site seeking introductory information about various tasks and capabilities.

With this in mind, I want to delve further into open source for those who are new to it. As popular as open source is on this platform, not everyone in the IBM i world is there yet. Due to a lack of time – or maybe even a lack of interest – there remain administrators who don’t use it and environments that don’t deploy it.

It’s a fairly simple path from Unix to Linux to Solaris to AIX to open source software running on IBM i. Of course syntax and technique differ, but many of the concepts easily move from there to here. Certainly compared to, say, managing Windows environments vs. IBM i, there’s much more common ground with open source.

AIX has had a Linux application toolbox for years. It consists of open source code compiled to run on AIX that can be installed as rpm packages. Of course it wasn’t always this way (as Alex Woodie explains), but these days, the open source model on IBM i is quite similar. The software installation dependency problem (a.k.a. RPM hell) has been addressed with Yum, which makes installing packages a breeze. Run a shell script and download a minimal amount of RPM packages to get started. Then either allow your machine to connect to the internet for additional downloads as needed, or set up a repository on a machine that RPM can access in the local network.

I’m also intrigued by the new capabilities that bash has received on IBM i. In the aforementioned article on Access Client Solutions, I left off with a SSH session connected to IBM i. Now, for those wishing to explore the world of open source, let’s get Yum and Bash working on the system.

I followed these directions. Different Yum install options are available. I chose the bootstrap.sql method.

I downloaded bootstrap.sql to my machine and selected the Run SQL Scripts option under the database section of the ACS menu.

It installed as expected, as seen below.

Be sure to read the whole document. Toward the end you’ll find some important notes to help you adjust your PATH. There’s also a link to a Yum cheat sheet.

Once this was set up and I was logged in via SSH to the command line, I located the Yum installation by running:

find / -name yum –print

That provided the full path to the Yum command:

/QopenSys/pkgs/bin/yum

I then ran /QOpenSys/pkgs/bin/yum update:

bash-4.4$ /QOpenSys/pkgs/bin/yum update
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package bash.ppc64 0:4.4-2 will be updated
---> Package bash.ppc64 0:4.4-5 will be an update
---> Package ca-certificates.noarch 0:2_git20170807.10b2785-1 will be updated
---> Package ca-certificates.noarch 0:2_git20170807.10b2785-2 will be an update
---> Package libcurl4.ppc64 0:7.70.0-1 will be updated
---> Package libcurl4.ppc64 0:7.70.0-3 will be an update
---> Package libsqlite3-0.ppc64 0:3.19.3-2 will be updated
---> Package libsqlite3-0.ppc64 0:3.32.3-1 will be an update
---> Package libssh2-1.ppc64 0:1.9.0-2 will be updated
---> Package libssh2-1.ppc64 0:1.9.0-3 will be an update
---> Package libutil2.ppc64 0:0.8.1-1 will be updated
---> Package libutil2.ppc64 0:0.9.1-1 will be an update
---> Package python2.ppc64 0:2.7.18-3 will be updated
---> Package python2.ppc64 0:2.7.18-5 will be an update
--> Processing Dependency: update-alternatives for package: python2-2.7.18-5.ppc64
--> Processing Dependency: update-alternatives for package: python2-2.7.18-5.ppc64
---> Package python2-rpm.ppc64 0:4.13.1-2 will be updated
---> Package python2-rpm.ppc64 0:4.13.1-7 will be an update
---> Package rpm.ppc64 0:4.13.1-2 will be updated
---> Package rpm.ppc64 0:4.13.1-7 will be an update
--> Processing Dependency: curl for package: rpm-4.13.1-7.ppc64
---> Package yum.noarch 0:3.4.3-18 will be updated
---> Package yum.noarch 0:3.4.3-19 will be an update
--> Running transaction check
---> Package curl.ppc64 0:7.70.0-3 will be installed
---> Package update-alternatives.ppc64 0:1.19.7-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                 Arch       Version                       Repository
                                                                           Size
================================================================================
Updating:
 bash                    ppc64      4.4-5                         ibm     2.1 M
 ca-certificates         noarch     2_git20170807.10b2785-2       ibm      12 k
 libcurl4                ppc64      7.70.0-3                      ibm     403 k
 libsqlite3-0            ppc64      3.32.3-1                      ibm     1.7 M
 libssh2-1               ppc64      1.9.0-3                       ibm     327 k
 libutil2                ppc64      0.9.1-1                       ibm      17 k
 python2                 ppc64      2.7.18-5                      ibm      26 M
 python2-rpm             ppc64      4.13.1-7                      ibm     276 k
 rpm                     ppc64      4.13.1-7                      ibm     2.2 M
 yum                     noarch     3.4.3-19                      ibm     1.2 M
Installing for dependencies:
 curl                    ppc64      7.70.0-3                      ibm     116 k
 update-alternatives     ppc64      1.19.7-1                      ibm      84 k

Transaction Summary
================================================================================
Install       2 Packages
Upgrade      10 Packages

Total download size: 34 M
Is this ok [y/N]: y
Downloading Packages:
(1/12): bash-4.4-5.ibmi7.2.ppc64.rpm                        | 2.1 MB  00:01
(2/12): ca-certificates-2_git20170807.10b2785-2.ibmi7.2.noa |  12 kB  00:00
(3/12): curl-7.70.0-3.ibmi7.2.ppc64.rpm                     | 116 kB  00:00
(4/12): libcurl4-7.70.0-3.ibmi7.2.ppc64.rpm                 | 403 kB  00:00
(5/12): libsqlite3-0-3.32.3-1.ibmi7.2.ppc64.rpm             | 1.7 MB  00:00
(6/12): libssh2-1-1.9.0-3.ibmi7.2.ppc64.rpm                 | 327 kB  00:00
(7/12): libutil2-0.9.1-1.ibmi7.2.ppc64.rpm                  |  17 kB  00:00
(8/12): python2-2.7.18-5.ibmi7.2.ppc64.rpm                  |  26 MB  00:13
(9/12): python2-rpm-4.13.1-7.ibmi7.2.ppc64.rpm              | 276 kB  00:00
(10/12): rpm-4.13.1-7.ibmi7.2.ppc64.rpm                     | 2.2 MB  00:01
(11/12): update-alternatives-1.19.7-1.ibmi7.2.ppc64.rpm     |  84 kB  00:00
(12/12): yum-3.4.3-19.ibmi7.2.noarch.rpm                    | 1.2 MB  00:00
--------------------------------------------------------------------------------
Total                                           1.9 MB/s |  34 MB     00:18
Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : bash-4.4-5.ppc64                                            1/22
  Updating   : libutil2-0.9.1-1.ppc64                                      2/22
  Updating   : libssh2-1-1.9.0-3.ppc64                                     3/22
  Updating   : libcurl4-7.70.0-3.ppc64                                     4/22
  Installing : curl-7.70.0-3.ppc64                                         5/22
  Updating   : rpm-4.13.1-7.ppc64                                          6/22
  Updating   : libsqlite3-0-3.32.3-1.ppc64                                 7/22
  Installing : update-alternatives-1.19.7-1.ppc64                          8/22
  Updating   : python2-2.7.18-5.ppc64                                      9/22
update-alternatives: using /QOpenSys/pkgs/bin/python2.7 to provide /QOpenSys/pkgs/bin/python (python) in auto mode
  Updating   : python2-rpm-4.13.1-7.ppc64                                 10/22
  Updating   : yum-3.4.3-19.noarch                                        11/22
  Updating   : ca-certificates-2_git20170807.10b2785-2.noarch             12/22
  Cleanup    : yum-3.4.3-18.noarch                                        13/22
  Cleanup    : python2-rpm-4.13.1-2.ppc64                                 14/22
  Cleanup    : rpm-4.13.1-2.ppc64                                         15/22
  Cleanup    : python2-2.7.18-3.ppc64                                     16/22
  Cleanup    : libcurl4-7.70.0-1.ppc64                                    17/22
  Cleanup    : ca-certificates-2_git20170807.10b2785-1.noarch             18/22
  Cleanup    : bash-4.4-2.ppc64                                           19/22
  Cleanup    : libssh2-1-1.9.0-2.ppc64                                    20/22
  Cleanup    : libsqlite3-0-3.19.3-2.ppc64                                21/22
  Cleanup    : libutil2-0.8.1-1.ppc64                                     22/22

Dependency Installed:
  curl.ppc64 0:7.70.0-3           update-alternatives.ppc64 0:1.19.7-1

Updated:
  bash.ppc64 0:4.4-5          ca-certificates.noarch 0:2_git20170807.10b2785-2
  libcurl4.ppc64 0:7.70.0-3   libsqlite3-0.ppc64 0:3.32.3-1
  libssh2-1.ppc64 0:1.9.0-3   libutil2.ppc64 0:0.9.1-1
  python2.ppc64 0:2.7.18-5    python2-rpm.ppc64 0:4.13.1-7
  rpm.ppc64 0:4.13.1-7        yum.noarch 0:3.4.3-19

Complete!
bash-4.4$

Again, referring to this article, I was able to run some of the commands the author suggests:

bash-4.4$ getjobid
Process identifier 233 is 003275/QSECOFR/QP0ZSPWP
bash-4.4$ cl
cl: usage: cl [-beEhiIkKnOpqsSv] COMMAND [ARG ...]
bash-4.4$ liblist
QSYS        SYS
QSYS2       SYS
QUSRSYS     SYS
QSHELL      PRD
QGPL        USR
QTEMP       USR

While working on this piece, there was a bash update. So I brought my system to the latest and greatest by running another Yum update command:

bash-4.4$ yum update
ibm                                                         | 3.6 kB  00:00
ibm/primary_db                                              | 372 kB  00:00
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package bash.ppc64 0:4.4-5 will be updated
---> Package bash.ppc64 0:4.4-6 will be an update
---> Package libncurses6.ppc64 0:6.0-6 will be updated
---> Package libncurses6.ppc64 0:6.0-7 will be an update
---> Package libopenssl1_1.ppc64 0:1.1.1g-1 will be updated
---> Package libopenssl1_1.ppc64 0:1.1.1i-1 will be an update
---> Package ncurses-terminfo.ppc64 0:6.0-6 will be updated
---> Package ncurses-terminfo.ppc64 0:6.0-7 will be an update
---> Package perl.ppc64 0:5.24.1-1 will be updated
---> Package perl.ppc64 0:5.24.1-2 will be an update
--> Finished Dependency Resolution

Dependencies Resolved

================================================================================
 Package                   Arch           Version            Repository    Size
================================================================================
Updating:
 bash                      ppc64          4.4-6              ibm          2.1 M
 libncurses6               ppc64          6.0-7              ibm          1.5 M
 libopenssl1_1             ppc64          1.1.1i-1           ibm          2.0 M
 ncurses-terminfo          ppc64          6.0-7              ibm          583 k
 perl                      ppc64          5.24.1-2           ibm           28 M

Transaction Summary
================================================================================
Upgrade       5 Packages

Total download size: 35 M
Is this ok [y/N]: y
Downloading Packages:
(1/5): bash-4.4-6.ibmi7.2.ppc64.rpm                         | 2.1 MB  00:01
(2/5): libncurses6-6.0-7.ibmi7.2.ppc64.rpm                  | 1.5 MB  00:00
(3/5): libopenssl1_1-1.1.1i-1.ibmi7.2.ppc64.rpm             | 2.0 MB  00:01
(4/5): ncurses-terminfo-6.0-7.ibmi7.2.ppc64.rpm             | 583 kB  00:00
(5/5): perl-5.24.1-2.ibmi7.2.ppc64.rpm                      |  28 MB  00:16
--------------------------------------------------------------------------------
Total                                           1.8 MB/s |  35 MB     00:19
Running Transaction Check
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating   : ncurses-terminfo-6.0-7.ppc64                                1/10
  Updating   : libncurses6-6.0-7.ppc64                                     2/10
  Updating   : bash-4.4-6.ppc64                                            3/10
  Updating   : libopenssl1_1-1.1.1i-1.ppc64                                4/10
  Updating   : perl-5.24.1-2.ppc64                                         5/10
  Cleanup    : bash-4.4-5.ppc64                                            6/10
  Cleanup    : libncurses6-6.0-6.ppc64                                     7/10
  Cleanup    : ncurses-terminfo-6.0-6.ppc64                                8/10
  Cleanup    : libopenssl1_1-1.1.1g-1.ppc64                                9/10
  Cleanup    : perl-5.24.1-1.ppc64                                        10/10
Updated:
  bash.ppc64 0:4.4-6                     libncurses6.ppc64 0:6.0-7
  libopenssl1_1.ppc64 0:1.1.1i-1         ncurses-terminfo.ppc64 0:6.0-7
  perl.ppc64 0:5.24.1-2
Complete!

I mentioned that bash update to show you why it’s important to regularly check for yum updates. The latest versions provide new features and enhanced security. Putting off system updates can be costly, but yum makes it pretty simple to keep current.

At this point, you should have OpenSSH, Yum, and the latest version of bash installed (as well as ACS). Now you’re set up to install other open source packages.

To find out what’s available, run “Yum list.” At the time of this writing there were 398 selections. Note: This doesn’t mean there are nearly 400 programs. Some of these packages are actually components that are needed to get various programs to run. The good news is that yum sorts out the dependencies – and you do have many packages from which to choose.

If you’re new to IBM i but are familiar with Linux, running things in bash while logged in via putty is great. Rather than having to login to a 5250 console and navigate green screens, you’re dealing with familiar commands and syntax. You can write scripts as you would with any other Unix-like system. And for experienced IBM i users, this interface also allows you to easily leverage what you already know.

For example, if I preface a command with “cl” in Bash, I can do some interesting things:

cl wrkactjob

As you can imagine the output from wrkactjob is long, so try it on your system. Some smaller output comes when I run:

cl wrkoutq

bash-4.4$ cl wrkoutq
CPD000D: Command *LIBL/WRKOUTQ not safe for a multithreaded job.
 5770SS1  V7R4M0  190621            Work With All Output Queues          2/12/21  10:13:03 CST            Page    1
 Queue        Library       Files    Writer       Status
 QDKT         QGPL              0                  RLS
 QPFROUTQ     QGPL              0                  RLS
 QPRINT       QGPL             14                  RLS
 QPRINTS      QGPL              0                  RLS
 QPRINT2      QGPL              0                  RLS
 QSPRCLOUTQ   QRCL              0                  RLS
 QSRVMON      QSERVICE          0                  RLS
 QS9SRVAGT    QSRVAGT           0                  RLS
 QEZDEBUG     QUSRSYS           0                  RLS
 QEZJOBLOG    QUSRSYS           1                  RLS
 QTPPPOUTQ    QUSRSYS           0                  RLS
         * * * * *   E N D   O F   L I S T I N G   * * * * *

You may need to get the hang of formatting commands, for example:

bash-4.4$ cl wrkoutq outq\(qgpl/qprint\)

CPD000D: Command *LIBL/WRKOUTQ not safe for a multithreaded job.
 5770SS1 V7R4M0 190621         Work With Output Queue        QPRINT      in  QGPL        2/12/21 10:14:04 CST            Page    1
 File       User       User Data  Status Pages Copies Form Type  Pty File Number   Job        Number Date     Time
 QPJOBLOG   QSYS       QLPSVR      RDY       1     1  *STD        5          1     QLPSVR     000135 02/26/20 02:45:05
 REQUESTROL QSECOFR    SQL         RDY      47     1  *STD        5          1     DSP01      000146 02/26/20 03:25:50
 QPRINT     QSYS                   RDY       1     1  *STD        5          1     QSLPSVR    000402 02/27/20 03:29:03
 QPRINT     QSYS                   RDY       1     1  *STD        5          1     QSLPSVR    000734 03/03/20 06:08:45
 QPRINT     QSYS                   RDY       1     1  *STD        5          5     QPRTJOB    000551 06/04/20 15:44:07
 QPRINT     QSYS                   RDY       1     1  *STD        5          6     QPRTJOB    000551 06/04/20 15:44:18
 QPRINT     QSYS                   RDY       1     1  *STD        5          7     QPRTJOB    000551 06/04/20 15:44:28
 QPRINT     QSYS                   RDY       1     1  *STD        5          8     QPRTJOB    000551 06/04/20 15:44:39
 QPCSMPRT   QSECOFR                RDY       2     1  *STD        5          1     DSP01      002556 07/02/20 09:13:15
 QPCSMPRT   QSECOFR                RDY       2     1  *STD        5          2     DSP01      002556 07/02/20 09:13:16
 QPCSMPRT   QSECOFR                RDY       1     1  *STD        5          3     DSP01      002556 07/02/20 09:13:16
 QPRINT     QSYS                   RDY       1     1  *STD        5          1     QSLPSVR    001271 07/07/20 02:31:27
 QSYSPRT    QSECOFR    CHGLICINF   RDY       1     1  *STD        5          1     DSP01      002826 01/21/21 12:39:35
 
         * * * * *   E N D   O F   L I S T I N G   * * * * *

Here’s another nice command:

cl dspmsg msgq\(qsysopr\)

Incidentally, running this example alerted me to an issue I needed to dig into:

CPI096E  99  INFO         Disk unit connection is missing.

Now if I want to install Python, PHP, or Ruby, I can just issue the “Yum install” command.

Finally, note the various support options for open source users on IBM i. Bitbucket allows you to tap into the broader open source community, and IBM i open source team members do monitor the site. If you can’t find what you need there, you can contact IBMers Camilla Sharpe and Jesse Gorzinski. We can help you do that if you need it.

Rob McNelly is a senior Power Systems solutions architect doing pre-sales and post-sales support for Meridian IT, headquartered in Deerfield, Illinois. McNelly was a technical editor for IBM Systems Magazine, and a former administrator within IBM’s Integrated Technology Delivery and Server Operations division. Prior to working for IBM, McNelly was an OS/400 and IBM i operator for many years for multiple companies. McNelly was named an IBM Champion for Power Systems in 2011, an IBM Champion Lifetime Achievement recipient in 2019, and can be reached at rob.mcnelly@gmail.com.

Edit: Have you upgraded yet?

Originally published by TechChannel March 19, 2021

New POWER9 servers running new firmware won’t work with your old HMC. Rob McNelly explains why, and provides some upgrade options.

Lifetime IBM Champion Jaqui Lynch recently wrote about why you need to upgrade to the latest HMC levels and how you can accomplish this task. Her article is full of good information, and you’ll find some great, bookmark-worthy links at the end.

But here’s the thing: We―and by we, I mean consultants, along with IBM itself―have been pounding this drum for a while now. Details about the HMC upgrade and end of life for early, x86 “appliance only” HMC levels have been public literally for years. I first noted the big changes in 2017, and I’ve kept writing about them.

Given all the information that’s been out there for so long, no one should be caught off-guard by IBM’s HMC plans. And yet, many seem surprised. Why? I believe many clients ignored the messaging because they erroneously assumed they could continue to upgrade software levels for their existing HMC x86 hardware indefinitely, as needed. Provided the performance is good, they don’t think about a replacement. And in environments where HMC use is limited to getting systems up and running, even slow performance might not spur taking action.

But it’s time. Getting current with the HMC can be put off no longer. Standing pat now comes with actual, real-world consequences. For instance, I know of a client that ordered a new POWER9 server. It arrived in the data center running FW950 firmware. This caused a bit of a headache as they were unable to connect their new server to their older HMC. What should have been a routine hardware refresh ended up being much more challenging, to the point where budgets and project plans were impacted.

Upgrade Options

So, one more time, here’s the official word from IBM support:

“POWER9 servers with FW950 require minimum HMC level V9R2M950. Starting November 23, 2020, IBM manufacturing began shipping all POWER9 servers with FW950. It is customary for each server firmware release to require a minimum HMC level. In the case of FW950 the minimum HMC level is V9R2M950. Managing a server with FW950 from an HMC lower than the required level will display “Version Mismatch” for the system state indicating the HMC needs to be upgraded to V9R2M950.

On April 24, 2018, it was announced that update support for x86-based HMC hardware appliances would end in 2018. V9R2M950 has dropped support of the x86-based physical HMC appliances (7042-CR9, 7042-CR8, 7042-CR7) whose last supported release is therefore V9R1.

What this means is that if the customer does not have a 7063-CR1 or Virtual HMC V9 appliance, they cannot HMC install/manage any POWER9 server running FW950.” (Emphasis mine)

It’s that simple: New POWER9 servers running new firmware won’t work with your old HMC. Moving forward is imperative. So how does one do that? Let’s run through the process of getting a new HMC. First, here’s another set of options for HMC upgrades, via either the network, a DVD or a USB flash drive. This document explains configuration and lists the default passwords you should change.

Speaking of change, note that there’s a new tool for managing that shiny new 7063-CR1 HMC. It’s called the baseboard management controller (BMC). This document tells you how to configure the BMC IP address. Accessing the HMC via the BMC gives you additional options compared to the x86 appliances of yesterday. For starters, you can get a console and power on the HMC remotely, which can be extremely useful in a lights out environment. You’ll know that the HMC is booting properly just by watching the output scroll by on the console. That’s reassuring when you’re performing work remotely. Should you require additional access control, this document explains how to restrict BMC access. And here’s how you can configure the BMC from petitboot or from the enhanced GUI. And here’s a list of other links on BMC management. Finally, this doc covers troubleshooting.

Change is inevitable. We’ve progressed from the classic GUI to the enhanced GUI. Now we’re advancing to new HMC hardware. I mean, who would have thought that we’d be spinning up virtual HMCs in VMware, yet clients are increasingly making that choice. On that note, call me old school, but I still prefer to see at least one physical HMC in an environment. If you’re experiencing an unplanned outage or downtime and need HMC access in order to troubleshoot or power on frames and LPARs, you’ll have a pretty tough time getting to the vHMC if VMware is also having an issue, or if the Power Systems server hosting your vHMC is having issues. Sure, that’s an unlikely scenario, but going virtual is a choice that should only be made once you’ve seriously considered all options and weighed the risks versus the benefits.

The Importance of Keeping Current 

If you’ve already upgraded, good for you. But if you’re just now realizing that you need to take action, I’d encourage you first to get on board with the latest HMC hardware, and then to think even bigger.

I don’t view this as an isolated issue. Updates are critical. We must regularly update firmware, update and upgrade the VIO server, and upgrade the OS. When we fall behind on any of them, we run the risk of being in an unsupported state. Sometimes you’re so out of date you risk running very old firmware and VIO code alongside brand new HMC code, and some of these code combinations may be either unsupported or untested. Why take those risks? Why put your back against the wall? As shown with the client story I shared at the beginning of this article, when you only make changes when forced to, you often make things more difficult for yourself. It’s so much better to stick to a regular maintenance schedule to keep your systems current and avoid unnecessary technical debt.

Edit: This article is an example of how pictures are worth thousands of words

Originally published by IT Jungle March 1, 2021

I enjoy reading about IBM i Fresh Faces. Sure, it’s refreshing to see that it’s not just graybeards like me who are working on the platform, but what really matters is that young people are learning about, getting hands-on with, and coming to love IBM i. I appreciate that many of these stories revolve around system/application modernization and open source solutions. This gives newcomers to IBM i a degree of comfort by reducing their learning curve.

Learning any new technology is challenging. Specifically, how do you go from learning about the concepts to actually getting on a machine and discovering what it can do? Of course, this is compounded in the world of IBM Power Systems hardware, where system access is often tough to come by for students and hobbyists. But for now, I’d like to provide some guidance for those newcomers who have access but aren’t sure where to start.

Recently I took note of a Webex replay on IBM i Access Client Solutions (ACS). I assumed that interested readers would all have a basic understanding of ACS — but that wasn’t the case. Shortly after this article was posted, I was asked how to actually get started using the tool. So let’s back up a bit. If you’re new to IBM i, or ACS specifically, register at IBM.com. You’ll need an ID to be able to access certain educational and technical documents, like this download page for ACS.

Once you’re logged in, you’ll see this page:

Check “I agree,” then click “I confirm.” This brings you to the page where you can actually download the code.

I selected the IBM i Access Client Solutions file (the top of this list) and downloaded it. I unzipped the file and it installed on the Windows machine I was using.

I tried to run it, but I didn’t have Java installed. There are many ways to get Java; I followed this tutorial.

I installed the code and set up my Environment Variables as instructed.

I verified that Java was installed by running:

java –version

I got back:

openjdk version “11” 2018-09-25
OpenJDK Runtime Environment 18.9 (build 11+28)
OpenJDK 64-Bit Server VM 18.9 (build 11+28, mixed mode)

After that it was a simple matter of clicking on the ACS icon and starting the app.

Again, I’m assuming you have the IP address and userid/password information of a system you can log on to. Like this:

If you cannot read the screen shot above, it says:

“Welcome to IBM i Access Client Solutions
IBM i Access Client Solutions provides a platform independent interface which consolidates the most common tasks for using and managing your IBM i system. Additional information about each task is available by either moving the cursor over the task or by using the tab and arrow keys to navigate between groups and tasks. To select a task, click on the task or use the tab and arrow keys to navigate to a task and then press the enter key.”

To get started, add a system configuration for each IBM i system you want to use or manage. Do this by selecting System Configurations from the Management tasks.

As the instructions state, toward the bottom of the list there’s the “Management” section. The first item within that section is “System Configurations.”

Again, if you cannot read that text:

System Configurations provides an interface to create and manage system configurations for your IBM i systems. Use this task to create the system configuration for each IBM i system you plan to use or manage. System Configurations supports:

• creating new system configurations
• changing preferences (such as SSL or password prompting) for existing system configurations
• adding a console configuration to an existing system configuration or locating a console for a new system

Use this task to create system configurations prior to using some other tasks.

I clicked New, entered my system name, and clicked on the connection tab. I instructed it to always prompt for a username and password. I also specified an IP address rather than have it look one up.

I returned to the general tab and verified the connection. This is what I saw:

Because I hadn’t installed ssh on the IBM i system, the ssh service failed to connect.

Back to ACS I went. In the top section, labeled “General,” I was able to click on 5250 emulator.

This brought up a green screen.

I entered my user ID and password.

This brought up another green screen where I could log into the system.

From here I went to Define or Change the System (number 7 in the screen shot above), then to Work with Licensed Programs (option 2, below).

I chose option 11 Install licensed programs.

Now I could search for the software I wanted. Again, more assumptions: at this point you need either to know how to install software, or have someone do it for you. You also need the appropriate install media. The ability to determine which software is on which DVD would also help. Googling “openssh IBM i” led me to this link.

I checked my system and found that I needed to install 5773SC1 (below).

To determine which DVD I needed, I searched for “ibm i media labels and contents 7.4,” which returned a doc labeled GI1199350.pdf. The “Media labels and their contents” document points you to the DVD with the software you need to install. Multiple tables in the doc help you determine which of the five DVDs you’ll need to load the software from. The one I wanted was B_groupx_04.

More assumptions: I’m assuming you know how to load a DVD on your system. Whether you need to physically load media into a DVD or you’re managing your system virtually, it’s important to know how you’ll install the code.

In my case I was using VIOS, so I needed to make sure the DVD was available in my virtual media repository. Then I needed to map it so that my IBM i client LPAR could use it. With the screen shots below, I’m assuming you’re familiar with HMC and the VIO interfaces. Obviously your mileage may vary.

I selected my VIO servers.

Once I was able to see the VIO servers, I clicked on the media repository view.

I chose VIOS2, the server with the media repository.

After I selected it, then I selected “Manage Virtual Storage.”

This brought up the list of media. I chose the DVD I needed.

After selecting it, I modified the partition assignment, linking it to the LPAR I needed.

 

 

Then I logged into my 5250 terminal. I returned to the “Install Licensed Programs” option.

I added it to the list of products.

I selected opt01, Programs and language objects.

Once it was installed, I was able to start sshd.

At this point, ssh was running.

My ACS setup was complete. Now I could choose either to continue using my 5250 emulator or try to open an ssh terminal. This option is located in the General section, second from the end of the list.

My system already had putty loaded, so I was able to start my session with a simple click of “SSH Terminal.”

SSH Terminal launches an already installed SSH client (terminal emulator). With an SSH terminal, it’s easy to run IBM i commands, invoke things in the Portable Application Solutions Environment (PASE), and access open source tools and technologies (e.g. Python, Node.js, Git, etc.).

If you receive a connection error within the launched SSH client (e.g., \”connection refused\”) or if a window pops up but immediately vanishes, you may need to start the SSH daemon by running this CL command:

	STRTCPSVR *SSHD        (This requires the 5733-SC1 product)

I appreciate that ACS provides context-sensitive help. In this case it let me know which product I needed to install and how to start sshd once it was installed.

With this access, you can login and get started. In future articles I’ll share more how-to information for newcomers to IBM i.

Rob McNelly is a senior Power Systems solutions architect doing pre-sales and post-sales support for Meridian IT, headquartered in Deerfield, Illinois. McNelly was a technical editor for IBM Systems Magazine, and a former administrator within IBM’s Integrated Technology Delivery and Server Operations division. Prior to working for IBM, McNelly was an OS/400 and IBM i operator for many years for multiple companies. McNelly was named an IBM Champion for Power Systems in 2011, an IBM Champion Lifetime Achievement recipient in 2019, and can be reached at rob.mcnelly@gmail.com.

Edit: I really like articles that cover more than one topic

Originally published by TechChannel February 17, 2021

Rob McNelly explains why older AIX versions run slower, and why it’s time to move on from AIX 7.1

Upon checking the error log, a client noticed errors that were pointing to a fiber port on their system. While the port physically existed on the machine, a fiber cable wasn’t connecting that port to a switch. The client acknowledged the port could be needed at some point, but for now they didn’t want it to logically exist. And naturally, they wanted the errors to stop.

The solution is provided in this IBM Support document:

“Usually when there are unused ports on FC Adapters, it is possible to disable those ports… and stop cfgdev/cfgmgr from configuring the devices… [This] will stop all the error log messages.

The steps and information provided… are intended to disable ports that are not connected and are intended to be not connected… [If] the errors are logged against ports that should be connected or should be in the “Available” status, please troubleshoot those adapters accordingly.”

The doc covers four methods for disabling a port, depending on how the card is configured. If it’s allocated directly to an LPAR, run the procedure that’s described for the root user. Alternatively, you can run the procedure using smitty, or you can script it. Finally, if you’re in a VIO server, you can run the following as padmin:

            Remove the fcs# device and all child devices.
            > rmdev -dev fcs# -recursive -ucfg
            Set the fscsi# device to not autoconfigure.
            > chdev -dev fscsi# -attr autoconfig=defined
            fscsi# changed

            At a future date when you need to use the port, you can enable it;
            > chdev -dev fscsi# -attr autoconfig=available
            fscsi# changed
            Then run cfgdev to configure the devices.
            > cfgdev

The Benefits of Baseboard Management Controllers

Baseboard management controllers (BMCs) are becoming more common in Power Systems environments. They can be used to manage OpenPower systems like the LC921, and they also work with newer POWER-based HMC appliances:

“IBM Power Systems servers use a… BMC and the Intelligent Platform Management Interface (IPMI) for system service management, monitoring, maintenance, and control. The BMC also provides access to the system event logs (SEL). The BMC is a specialized service processor that monitors the physical state of the system by using sensors. A system administrator or service representative can communicate with the BMC through an independent connection. The BMC uses IPMI and is contained on the system backplane. IPMI provides one communication method to the BMC, by using a command line interface. IPMItool can be used either from a remote Linux system, or from the host operating system console window. IPMItool remote connections to the BMC can be done by using either the serial connection to the BMC, or through a configured Ethernet port. The BMC provides a web interface, which provides a graphical user interface (GUI) that can be accessed from a management console or workstation that has network connectivity to the BMC. This connection requires an Ethernet port to be configured for use by the BMC.”

If you have a POWER-based HMC appliance in a lights-out data center, or if you’re not currently traveling to data centers, powering on/off with a BMC is a handy option.

Replacing a Physical Disk in a VIOS Environment 

Have you had a physical disk fail in configurations where VIOS is booting from internal disk? Here’s a good procedure to follow if your disk replacement skills are rusty:

“Question
How to free up a failing disk that’s part of a PowerVM Virtual I/O Server, mirrored rootvg in preparation to replace the disk. This applies to VIOS 3.1.

Cause
Mirrored VIOS rootvg disk is failing.
Note: to determine if the disk may need to be replaced, contact your local Hardware Support Representative.”

Scroll to the bottom of that page, and you’ll find a link that explains how to mirror rootvg again once you’ve replaced the disk.

These concepts may seem basic, but many administrators come from non-AIX backgrounds (including IBM i) and are unfamiliar with these sorts of tasks. For anyone who hasn’t set up VIOS and doesn’t know how to manage it, these types of documents can be very helpful.

Managing VIOS Backups 

Here’s something many of us have been looking forward to for awhile: With HMC version 9.2.950, you can manage the I/O configuration of VIOS and backup VIOS images directly from the HMC. This could eliminate the need to have a NIM server to manage VIOS backup and restore operations, although—just as some people prefer belts and suspenders—you may be more comfortable doing both until your confidence grows with this new option.
 
Here are the steps involved: 
 
1. In the navigation area, click the HMC Management icon, and then select Templates and OS Images.
 
2. From the Templates and OS Images window, select the VIOS Images tab, and then click Manage Virtual I/O Server Backups.
 
3. In the Manage Virtual I/O Server Backups window, select the Virtual I/O Server Configuration Backup tab. A table is displayed that lists all the backup files of the VIOS configuration that is taken by the HMC. Additionally, you can view the time at which the configuration file was last edited.
   a) To take the backup of the input/output configuration of a VIOS, click Backup I/O configuration. In the Backup I/O configuration window, select the managed system and the VIOS for which the backup is created, and then specify a name for the backup file. The name you specify must consist of 1-40 characters including file extension .tar.gz. You can use the characters A-Z and a-z, the numbers of 0-9, the dot (.), the dash (-) and the underscore (_) characters.
   b) To rename an existing backup file that is stored in the HMC, select a configuration file from the table and click Action > Rename.
   c) To restore the VIOS input/output configuration, select a backup file which contains the I/O configuration of the VIOS that you want to restore, and click Action > Restore.
 
4. In the Manage Virtual I/O Server Backups window, click the Virtual I/O Server Backup tab. A table is displayed that list all the VIOS image backup that are taken in the HMC. Additionally, you can also view the name and size of the VIOS image, the time when the VIOS image file was last edited, the managed system and the VIOS from which the image was captured.
   a) To take the backup of the VIOS image, click Create Backup. In the Create Backup window, select the managed system and the VIOS for which the backup is created, and then specify a name for the backup file. The name you specify must consist of 1-40 characters including file extension .tar. You can use the characters A-Z and a-z, the numbers of 0-9, the dot (.), the dash (-) and the underscore (_) characters.
   b) To rename an existing VIOS image backup file that is stored in the HMC, select a backup file from the table and click Action > Rename.
   c) To remove a VIOS image backup file from the HMC, select a backup file which contains the VIOS configuration that you want to remove from the table, and click Action > Remove.
 
5. Click OK.
 

x86 HMC Reaches End of Service

If you haven’t heard, it’s time to upgrade your x86 HMC:
 
“HMC V9 R1 is the last release to support the 7042 machine type. HMC V9R2 will support the 7063 machine type and Virtual HMC Appliances (x86/ppc64le) only.
 
Note: iFixes and Service packs for V9 R1 will be supported on 7042 machine types until EoS of V9 R1.”
 

A Quick Case for Upgrading to AIX 7.2 

In his AIXpert blog, Nigel Griffiths explains why AIX 7.1 users should upgrade to AIX 7.2:
 
“No sensible technical person would be running anything older than AIX 7.1 at this point in time, due to:

• Lack of support or support comes at a high price via Service Extensions
• Lack of security updates
• Missing advanced functions and advanced features that are only found in later AIX releases, particularly AIX 7.2
• Not making full use of POWER8 and POWER9 servers. For example, AIX 6.1 can only do SMT=1,2 and 4 (no SMT=8!).
• Missing years of development into removing serialization on locks and latches, adding parallel execution, shortening path lengths (less CPU cycles used to get work done), better performance tuning options, better out-of-box performance settings, and field hardening like trace and diagnostics.” 

Putting these all together: older AIX versions run slower.

Edit: these are still good skills to have

Originally published by TechChannel February 1, 2021

We may bash one another with our opinions, but every AIX admin needs to know vi and ksh

People are pretty passionate about the tech that they use. That’s hardly a news flash. It can be argued that none of us is truly objective, but I’ve been through enough heated discussions over time to understand that strong feelings exist about computing.

So we may as well cop to our biases. I’ll start. I freely admit to being an AIX bigot, and I know that colors how I see the world.

What about you? Do you prefer Windows or Linux or MacOS on your desktop? I would say that MacOS and Windows run into these issues less than Linux, but your mileage may vary. But is preference even the most important thing? If disparate desktop environments are being used, the simple and straightforward can become more complicated. Sure, workarounds are doable, but that eats up time that could be spent collaborating.

So would it be better that everyone use the same popular OS to maintain a friction-less environment? If I can get Zoom or Webex or Microsoft Teams running on a Windows machine, it’s a safe bet you can get it to work on your Windows machine. That means we can quickly get on a shared session and work together.

This sort of thing is hardly new. I worked at IBM in the early 1990s, when the company was exhorting us to eat our own dog food. That meant using Lotus Notes for our mail environment, along with Lotus Office Suite for word processing. As a result, the process of exchanging incompatible files with customers was typically challenging. I ended up requesting permission to run Microsoft Office products. It was the only way to resolve our issues, and once we were on the same software, our problems went away.

Of course the counter argument is: shouldn’t the quality of the product matter most? What about viruses and malware? That code is often being written to infect the masses that run Windows. The argument for running Linux or MacOS is they give you a better chance of avoiding those issues.

This extends to the enterprise. We hear about ransomware encrypting and locking Windows servers, primarily. I’ve not heard of similar things happening with AIX or IBM i running on POWER.

Of course we can drill way down here. Even among AIX users, strong preferences exist over which tools and commands to deploy and run in any given situation. For instance, what’s your favorite editor? Are you in the vi camp? Emacs? Do you prefer something with a GUI, or something like Pico? Just typing this out, it seems like such a minor consideration. And yet, the opinions are many, and the feelings are real.

What about your favorite shell?

A Unix shell is a command-line interpreter or shell that provides a command line user interface for Unix-like operating systems. The shell is both an interactive command language and a scripting language, and is used by the operating system to control the execution of the system using shell scripts.

Most experienced Linux users are pretty comfortable with the bash shell. But do you know you can install that shell on AIX? Use the chsh command, or change your shell to bash. But be careful:

AIX default shell is the Korn shell. There are quite some other shells one can use with AIX. However IMHO anything but ksh or ksh93 in AIX is about 99.xx% compatible only. The missing .xx% can cause you some headaches when you rather would concentrate on some other os related problem. Therefore, as an AIX administrator I’d always recommend to stick to ksh.

On that note, is there anything more frustrating than troubleshooting a problem and then discovering it was self-inflicted? Changing your root shell can lead to problems:

Question: Can I change root or another system account user’s default shell from ksh to ksh93, or bash?

Answer: The Korn shell (/usr/bin/ksh) is set up as the default shell. The default or standard shell refers to the shells linked to and started with the /usr/bin/sh command.

The AIX Operating System is tested with SHELL=ksh in all system account script processes, unless otherwise defined in the script. Using chsh or otherwise modifying system user accounts in /etc/passwd to change a system account shell to non-ksh could result in script failures at run time. Users are advised to test any default shell changes carefully in their environment.

Note: The bash shell is an open source product, and is not supported by AIX Support cases.

IMPORTANT: Do not replace /usr/bin/sh or /usr/bin/ksh with other binaries!

• The system is in 32-bit mode in phase 1.
• The kernel cannot load a 64-bit binary until phase 2 of the boot.
• There are shell scripts that run during phase 1 of a system boot.

# file /usr/bin/ksh
-> executable (RISC System/6000) or object module
# file /usr/bin/ksh93
-> 64-bit XCOFF executable or object module not stripped
# file /usr/bin/bash
-> 64-bit XCOFF executable or object module

To reiterate: Do not change root’s default shell!

I get it, you really like bash. You really don’t like ksh. Tough. You’re a professional. Learn how to navigate with ksh. Get so proficient at it that it becomes second nature. You need to master it. If you’re new to AIX, there are resources. This old book remains an excellent Korn primer. Here’s another older book. Same deal.

As I’ve said often, get on a system and get hands-on. Trial and error—while always trying to accomplish something, day in and day out—is the best way to learn in my experience. The time to learn is not when the system is down; that’s when you need to know. When a system is down and maintenance must be performed, an expert is needed: you. This is why you must be well-versed with the system’s built-in tools. If you cannot already run vi and edit a file without thinking about it, practice more. If you cannot log into ksh and navigate your shell history and edit commands on the fly, practice more.

Arguments can be made, sides can be taken, but my position is that knowledge of vi and ksh represents the bare minimum for being able to maintain an AIX system.

Edit: My third post for IT Jungle

Originally posted February 1, 2021

School’s been out for me for a very long time, but I still enjoy learning. I gain a sense of satisfaction whenever I learn something new. Specific to technology, exposure to new concepts helps me understand how things work together. I cannot count the number of times where I watched over someone’s shoulder, or watched someone on a shared screen, to learn about a new tool or technique, or a different way to set up my desktop or environment.

Watching and listening to people is my preferred way to learn, but other forms of education – reading IBM Redbooks and other documents, or articles, or watching webinar replays – are also worthwhile. Pick a topic in the Power Systems ecosystem, whether it is the pros and cons of a virtual HMC, how many physical adapters you can fit into a given machine, how to transfer files or backup machines, CL or RPG programming techniques, updates to the open source environment on IBM i or overall general best practices, I love to soak it all up.

Digging into new topics – even if it’s only an inch deep – is especially important for IT pros in smaller workplaces. At smaller companies, fewer people wear more hats, and they’re typically asked to do more.

Even if I don’t use this information immediately, even if I never use it, I still value the experience of learning. I like to know what’s possible. To me it’s worth the time to get exposed to the concepts, and the less familiar I am with something, the more motivated I am to read about it. You never know when some tidbit of information that you’ve absorbed will come in handy. Having even the slightest introduction to a topic makes it much easier to conduct a search or ask a question later on.

Scott Berkun offered a similar perspective on Twitter recently: “If you’re experienced in your job, a great way to grow is to study something else. Go read a book about or go to a conference relating to something you know little about. You’ll ask big questions. You’ll learn new models and thoughts. You’ll return to your work with fresh eyes.”

In that vein, many of you may be unfamiliar with the inner workings of the PowerVM Virtual I/O Server, a.k.a. VIOS. It’s worth learning more about this topic though, because action needs to be taken in virtualized environments.

VIOS allows you to virtualize Power Systems servers. While I see a great deal of it in AIX and Linux on Power environments, it also supports a growing number of IBM i workloads. If you’re a VIOS user, you should know that, as of October 2020, VIOS 2.x is no longer supported by IBM. And you should understand why it’s important to move VIOS 3.1.x as soon as possible.

The withdrawal of support for VIOS 2.x doesn’t mean that you’re completely out of options for getting support. As is the case with most withdrawn offerings, IBM will still provide extended support. But paying the premium isn’t the only issue, or even the primary issue.

Also, this isn’t a boilerplate plea to move to the latest and greatest. You should move because VIOS 3.x is fundamentally different from VIOS 2.x. The updated version is designed to function more efficiently with newer Power Systems hardware (Power8 and later). VIOS 3.x runs AIX 7.2 under the covers, whereas VIOS 2.x runs AIX 6.1. If you’re not well-versed in the AIX operating system numbering scheme, AIX 6.1 arrived in the 2007 timeframe; 6.1 TL9 went end of support in April 2017. So under the covers, those versions of VIOS were getting long in the tooth. The new VIOS code, as noted, is better able to exploit the improvements in the Power hardware, (think of being able to dispatch to more threads, etc.). In addition, IBM removed VIOS legacy code (does IBM Systems Director ring a bell?), resulting in overall performance improvements with VIOS 3.x.

As a refresher, storage that’s been allocated to a VIO server can be connected to a VIO client by using either NPIV or vSCSI. Each option has advantages and disadvantages, but these days I see NPIV more often than not.

The following description of a separation of duties is mostly applicable to larger shops, where different teams have different roles, verses a smaller shop where one guy may do it all. With NPIV, the SAN guys are zoning LUNs directly to the client LPAR, whereas with vSCSI they’re zoning LUNs to the VIO server itself. It’s an extra step for the Power Systems administrator to then map the allocated LUN to the client IBM i LPAR. SAN guys generally prefer the added visibility into which LPARs are using the actual LUNs. Rather than see a LUN disappear when it gets mapped to a black box (the VIO server), I find that they tend to prefer NPIV. In a smaller shop, since it is the same guy doing the zoning and the mapping, it may be less of an issue.

With the code changes that have occurred, it’s important to recognize that the process of upgrading to VIOS 3.x differs from what you’re accustomed to. You’re not simply putting on a fixpack or service pack; you’re doing an under-the-covers OS upgrade from AIX 6.1 to AIX 7.2. Upgrading is still fairly straight-forward, but it does require some planning and preparation, so approach it carefully.

The good news is that most environments set up VIO servers in a dual VIOS configuration, which provides multiple paths to storage and the network and allows for maintenance activities to occur without affecting running client LPARs. The idea is that you can upgrade VIO server 2, test it, and then fail everything over to that second server and then conduct maintenance on VIO server 1. Of course, you should subject your VIO failover process to regular testing. As with any high availability solution, or even system backups, if you do not test, you cannot be sure that things will work correctly when you need it to.

IBM has tools and methods to help you perform the upgrade, but the basic idea is you’re backing up your configuration, doing a new VIOS install, and then restoring the configuration information to your fresh VIOS copy. A friend has been performing quite a few of these upgrades lately. Although IBM has a tool you can use to backup your data, he prefers to gather the configuration data himself.

In any event, he recounted a unique experience. He’d been told that, in this particular environment, all of the IBM i clients were connected via NPIV to their storage. After gathering the necessary data and doing the fresh install of the VIOS code, he went ahead and restored the network and the NPIV configurations. He then failed over the clients to his newly built VIO server and began work on the other one. Most of the IBM i clients were just fine, though a few started spitting out LED codes, indicating that they had lost access to their disks. It turns out that some vSCSI disk connections remained in the environment after all. As noted, he’d taken the word of others regarding how the disk was connected rather than verify the connections himself.

What would you expect to happen if you pull all the paths to the disks out from under a running system? I’d certainly expect them to crash. However, once the problem was understood (this took about an hour), both VIO servers had their vSCSI mappings put back in place, and the LPARs just continued from where they left off. They didn’t crash, they didn’t reboot; they just continued running as if nothing had happened. The customer was able to login and verify the system looked and behaved as they expected it to.

Maybe that speaks to the resilience of IBM i, or maybe my friend got lucky. The maintenance was performed during a quiet time of the day, so the systems weren’t being taxed in that moment. Regardless, I was impressed.

But don’t take the wrong lesson from this story. Yanking disk paths away from running production systems willy nilly is never a good thing. However, I would like to try and reproduce this behavior in the lab. When discussing this result with others, I’ve heard skepticism. There must be another reason why that environment didn’t crash. One thought is that there must have been a connection to the load source disk via NPIV.

I intend to get to the bottom of it. I look at it as another learning opportunity. Formal school may be out for most of us, but we’re still learning.

Rob McNelly is a senior Power Systems solutions architect doing pre-sales and post-sales support for Meridian IT, headquartered in Deerfield, Illinois. McNelly was a technical editor for IBM Systems Magazine, and a former administrator within IBM’s Integrated Technology Delivery and Server Operations division. Prior to working for IBM, McNelly was an OS/400 and IBM i operator for many years for multiple companies. McNelly was named an IBM Champion for Power Systems in 2011, an IBM Champion Lifetime Achievement recipient in 2019, and can be reached at rob.mcnelly@gmail.com.

Edit: My second post for IT Jungle

Originally posted December 14, 2020

Those of a certain age will certainly remember the moment in Fast Times At Ridgemont High when Jeff Spicoli got himself into a bind after he wrecked his friend’s car. Luckily, Spicoli’s dad was a TV repairman, and he had an ultimate set of tools. Spicoli knew that with those tools, he could fix it. That level of confidence is intoxicating, although in this case it was possibly misplaced.

I have a friend that lives around the corner from me that actually does have the ultimate set of tools, and knows how to use them. For example, he recently rebuilt his Jeep from the frame up. The other day my son, a high school senior, reported to me that there was a sound coming from one of the front wheels of his car. I took it for a drive and the metal on metal grinding sounded expensive. My friend offered to take a look, quickly diagnosed the problem, went with us to the auto parts store and back to his garage where he replaced the worn-out brake pads that were squealing. When you combine tools with experience, people can accomplish a great deal.

You have to take care of the tools that you have, part of what made my friend so effective was that he kept his tools clean, he kept his garage organized, he put his tools away when he was done using them, he knew where his tools were, he practiced with them over time, and he knew how to use them. Although our tools may be digital in nature, the same principles apply.

There’s a relevant story from Steven Covey’s 7 Habits of Highly Effective People:

The lumberjack was trying to cut down a tree with and was swearing and cursing as he labored in vain.

“What’s the problem?” The man asked.

“My saw’s blunt and won’t cut the tree properly.” The lumberjack responded.

“Why don’t you just sharpen it?”

“Because then I would have to stop sawing.” Said the lumberjack.

“But if you sharpened your saw, you could cut more efficiently and effectively than before.”

“But I don’t have time to stop!” The lumberjack retorted, getting more frustrated.

The man shook his head and kept on walking, leaving the lumberjack to his pointless frustration.

As an IT consultant, ongoing education and training is the equivalent of sharpening my saw and keeping my tools organized. Over the years I’ve always made time to attend conferences. These days I take part in as many free online education sessions as I can. I try to listen to them when they’re live, but if I have a conflict when the session takes place, I download the slides and watch the session replays when I get time. Invariably, I’ll learn something new.

In our world, you may think you know it all – or at least you believe you know everything you need to know to do your job. But of course technology doesn’t stop: The systems and solutions we work with are always being updated. Then there’s the more mundane reality that sometimes you’ll pick up a tip for doing a particular task, and it might be months or years before you get a chance to put what you learned into practice. And guess what? You’ve forgotten. I certainly forget things I’ve learned on occasion. The point is relearning is also a part of learning. Reminders never hurt.

There are a number of excellent educational resources out there. But if you’re looking for a single resource that provides current technical information for Power Systems users, check into the Power VUG Technical Webinar Series:

“Power VUG (Power Virtual User Group) is a monthly technical webinar series for IBM Power Systems. The webinars are informal and have a focus on how-to, how-it-works, best practice, and hints and tips. Many include live demonstrations. They are relevant to AIX/IBM i/Linux on Power Systems. The series is aimed at a technical audience – operators, systems administrators, and technical specialists – those using/planning to use IBM’s Power Servers. It is open to Clients/Business Partners/IBMers.”

They’ve done more than 100 sessions since 2011, covering topics like VIOS, IBM i suspend/resume and Live Partition Mobility, performance, monitoring, configuration best practices, IBM i mobile access, and Ansible on IBM i, to name just a few. Power VUG webinars aren’t simply people reading off of slide decks. Many sessions include live demos, and most have Q&A sessions. There’s nothing like actually watching the tools in action to get a good feel for how they work.

As an example, I’ll highlight a session presented by Tim Rowe, Business Architect of Application Development for IBM i on November 11. I encourage you to download Rowe’s slide deck and watch the replay:

Slide 2 is a roadmap of the information Rowe covers: IBM i Access Client Solutions (ACS) v 1.1.8.6, IBM Navigator for i Performance Data Investigator, the Digital Certificate Manager GUI, IBM i services powered by SQL, Administration Runtime Expert, Nagios, and open source plugins. There’s a lot here.

The session begins by defining access management. Who is accessing the system? There are tools and access for everyday users, while those who manage the system have a different, more powerful set of tools. A DBA will require yet another set of tools to make sure the database is running as expected.

In today’s ecosystem we need to be sure the tools run anywhere. Mac, Linux, Windows, and various mobile devices, all need to work and be supported, and with the new software they are.

IBM i Access for Windows went end of life in April 2019; it has been replaced by ACS. There are two ways to get the package: either download it from here, or if you’re running IBM i 7.4 SF99662 or IBMi 7.3 SF99722, get it directly from your machine by pointing to your IFS at /QIBM/proddata/Access/ACS/Base.

Once you apply the PTFs, visit this location regularly to ensure you’re keeping current with ACS. Rowe also shows how with version 1.1.8.6 version of the software, you can modify ACS properties in ACS to ensure that your searches for updates are targeted to your local system. It’s important to keep ACS current. Customer input is important to the ongoing development of this product.

There’s now the option to select and upload multiple files to your IFS, along with the ability to upload an entire directory with its contents. In the past only a single file could be uploaded at a time. IBM has improved both the performance and the actual interface, and have provided a better capability to filter files in different ways. Other additions include integrated file unzip support and the capability to view ILE source members or EBCDIC files as UTF-8 text files. Finally, there’s the capability to run SQL scripts. Watch the replay and you’ll see examples of how this is helpful.

IBM i developers are making greater use of SQL. Scott Forstie, senior software engineer at IBM, has a GitHub page which contains 67 working examples. And COMMON has a video series on SQL. Also check out SQL Tutor.

A neat part of the demo involves content assist. You’re prompted for possible values that will make sense in the context of the command that you’re creating. Rather than needing to look up commands or syntax, you can click your mouse to see which options are available. There’s a context-sensitive interface that can help with CL and SQL; it’s also useful for configuring commands and options for the commands. The demo shows various ways you can build these commands, along with using the formatter tool. Click on the “insert from examples” button; if you don’t write SQL, you can use built-in examples and learn from them. The examples help answer questions about the system, like which files are owned by users, IFS growth, etc. This section in itself is a great reason to watch the replay.

IBM has also changed the ACS interface. The preferences and property settings are now in the same place, which makes it easier to keep track of the different options you might want to set instead of searching for options in multiple places.

Under tools, there’s an open source package manager. From here you can manage the rpm packages on the IBM i system. In the past, if your machine didn’t have direct access to the repository, you had to download all the packages before you could use any. Now, if your IBM i machine is isolated from the internet, you can use your workstation in conjunction with an SSH tunnel to automatically download from an rpm repository on the internet and get that copied to your IBM i system.

Moving away from ACS, there is a discussion around IBM Navigator for i. There’s a performance data investigator (PDI) that you can use with collection services, disk watcher, job watcher, database, etc. You can view the data in new graphical interfaces, and this new functionality makes it an important feature. Tons of system information is available for analysis. If you want to know how a specific chart was generated, can click on the show SQL button to see what was run and what data was gathered. Don’t forget: Navigator is a free solution that comes with the system. Again, watch the demo to see it in action.

A new digital certificate manager, available for IBM i 7.3 and 7.4, makes it much easier to manage your certificates. You can manage local certificates, you can look at all of the certificates that are on your local certificate store. You can get information, search, look for expired certificates, etc. There’s also an SQL service. While the older tool you used for your digital certificates remains available for now, expect it to eventually go away.

Administration Runtime Expert (ARE) is a tool for comparing central systems to endpoints. You can, for example, move PTFs around your environment using this powerful and free tool. It was stated during the session that it is deserving of its own session to cover all it can do.

Toward the end, learn more about monitoring your environment using Nagios. There are lists of supported elements like CPU utilization, disk status, list job information from a subsystem, message checks, etc. It’s also possible to use SQL plugins to define your own rules, your only limitation as to what you monitor is your imagination.

Why did I go into such detail on one webinar? Because I’ve written enough articles over the years to know many readers don’t invest the time to click on the links and listen to the replay. I wanted to try to whet your appetite. If you’re interested in learning more about the capabilities of IBM i and Power Systems hardware, Power VUG webinars are a great way to sharpen your saw. Please click that link and take some time to watch and learn, then you too will be able to add to your ultimate set of tools.

Rob McNelly is a senior Power Systems solutions architect doing pre-sales and post-sales support for Meridian IT, headquartered in Deerfield, Illinois. McNelly was a technical editor for IBM Systems Magazine, and a former administrator within IBM’s Integrated Technology Delivery and Server Operations division. Prior to working for IBM, McNelly was an OS/400 and IBM i operator for many years for multiple companies. McNelly was named an IBM Champion for Power Systems in 2011, an IBM Champion Lifetime Achievement recipient in 2019, and can be reached at rob.mcnelly@gmail.com.

Edit: My first post for IT Jungle

Originally posted November 30, 2020

There are philosophical differences that exist between people regarding the best ways to configure, manage, and maintain infrastructure, and this holds as true for infrastructure built around IBM i as it is for other kinds of platforms. The easiest and fastest thing to do – and what makes the most sense according to plenty of people – is to do what they have always done. Just keep replicating the past out into the future to infinity.

Others point out that this mindset stifles progress and ignores all of the innovations that have come to the IBM i platform through the years. How can you even consider not making the full use of your computer’s capabilities? Why are you still running your system the way you did in 2010, or in 2000 or even 1990?

A recent tweet by Larry Bolhuis sums up this way of thinking nicely, if a little hyperbolically:

“When you update your @IBMPowerSystems are you truly updating or just replacing one generation of CPU w/ the next and adding a dusting more disk and memory? If your business partner didn’t discuss PowerVM, SAN Storage, and VTLs before you bought, get a new partner.”

To expand on that, why would you ignore the power and flexibility to be gained by connecting to a SAN? Among many other things, SAN-based replication, snapshots, the ability to perform Live Partition Mobility, are given up when you refuse to move forward and think differently. Why purchase internal disk or an SSD that’s only available to one system in your environment when you could share in a pool of disk that’s used by other machines?

IBM resellers and business partners should strive to educate customers, and increase their own awareness. They should point out the advantages of, for instance, connecting to a SAN. Of course to assume that all IBM i installations even have SANs would be a mistake. Sure, they are present in most large environments, but they may still be the exception in smaller shops. One size does not fit all, and real-world circumstances must be recognized and acknowledged. Still, sellers should be sharing this information with you. Small shops may still benefit from connecting to a smaller IBM SAN, for instance. What’s important is that these discussions occur and customers learn of new options.

It’s a fine line, of course. Some choose to stay put. They’re satisfied, in many cases thrilled, with what they have. The system just runs. There have been no outages, and what more can you ask for from an investment in hardware? Everything is simple. Troubleshooting is straightforward; diagnoses are easy. The focus is the machine and its components, and possibly the network connection. Personally, whether it’s in my own environment or through my consulting with clients, I always get a charge out of working with new technology and getting the most functionality out of it that I can. But I can see the elegance in having a reliable standalone system, and in that way of thinking.

When it’s time for a hardware refresh, the request in this scenario is simple, something along the lines of: “Please give me the current equivalent of what I’m currently running.” Maybe a little more disk is needed, or a faster, higher-capacity tape drive. With the speed of today’s processors even a factory de-configuration of some cores is a consideration; if the workload in question is small enough, maximum horsepower may not be needed. So the order is placed and the upgrade is performed. You are content, knowing that this new configuration will last for years, or at least until this hardware reaches end of life and you’re compelled to do another upgrade.

I was recently part of an interesting back and forth with Bolhuis and others about these sorts of scenarios and choices. It was noted that customers are not always educated about what they can do with their Power hardware. They may not fully appreciate the benefits of virtualization, or running things differently than they always have:

“You know your @IBMPowerSystems can run more than just #IBMi or just #AIX! Use your next upgrade to enable deployment of both, or Linux, or all three. Don’t let yourself be boxed in by old single system thinking. Be sure your BP knows the options for virtualization!”

Especially for those of us who think about this stuff for a living, it’s natural to assume that everyone else is just as engaged. But not everyone attends virtual briefings or consumes every detail of IBM’s announcements, they have better things to worry about. (That is why you have The Four Hundred doing this for you, in fact.)

Replacing existing hardware with similar, more current hardware may make perfect sense, but remember it isn’t the only way. Business partners should be partners. They need to really listen to you, they should strive to understand your needs, answer questions and keep you informed. They should help you solve your business problems. And people like me aren’t serving our clients and customers if we don’t suggest alternatives. With some slight modifications, a traditional standalone system can host multiple LPARs, including Linux on Power workloads, and production and development LPARs, for example.

A position we do not want to be in is to get a request from management or from system users that we cannot fulfill. They may come to us and say that they understand that in theory they can add an LPAR to an existing Power System. Or they may tell you that they are under the impression that they can boot the system from a SAN. They may approach you and tell you that they understand that they can do X, Y, or Z.

The answer may very well be: “Yes you can, in theory, but since we did not discuss this, you’re missing an HMC, and you’re missing these other components that will enable that functionality. So until we remedy the situation, the answer is going to be no.”

None of this is a revelation; these capabilities have existed for years. For you the decision may not lie in knowing that you can do it, but in the incremental costs involved, or the skills available, or the thinking that if it isn’t broke don’t fix it, or as stated before the elegance of the system running as it is.

Some may view IBM i as a legacy environment and seek an excuse to discard it for something new, shiny and sexy. Education needs to occur at the management and executive levels as well. Power should be a strategic platform to your business, and wherever and however possible, it should be exploited to its full potential.

I think we can agree that some customers, and maybe even some sellers in the IBM i world aren’t up to speed — or simply not all that interested in learning about everything that the platform can do. That’s unfortunate, but that’s part of my takeaway from the discussion with Bolhuis.

Blame can go in all directions on this. Should IBM i operators and admins make more of an effort to seek out the information? Should IBM make information easier to find? Should employers be providing their employees with the training and classes and conferences they could benefit from? Should sellers be more active in educating themselves, and having conversations and whiteboard sessions and lunch and learns with customers?

It could simply be a product of the reality we all live in. Everyone is doing more, everyone is busy. Day to day it is easy to let learning take a back seat to current system issues. And that doesn’t even begin to get into the unprecedented chaos that has been 2020. With our day to day jobs, with our lives, it’s tough to balance everything. It all takes time, and there are only so many hours in a day.

Ultimately though, a lack of education or knowledge or experience can be remedied. The discussions are good – they’re imperative, really. Even internal discussions can be beneficial. Maybe ask yourself: How would life change if you went from a single production LPAR to having access to multiple instances of IBM i? Yes, there are costs to consider: licensing, additional cores, additional memory, additional disk, etc. Some of you will look at the costs and dismiss this proposition out of hand. But sometimes these decisions are based more on valuing the way things have always been done and fearing the unknown.

Getting a test machine is a good start when it comes to trying new things. No one wants to mess with production, so a test box or partition is the best way to try something new or get better at what you do. Implementing a dedicated environment where you can change code and test backups, upgrades, installs, and so forth and never have to worry about affecting the business is really a no-brainer decision.

So let’s initiate these conversations. What would make your life easier? Where would you benefit from taking a slightly different approach? If you could change anything, what would you do differently? Needing to make no changes at all is also an acceptable answer, provided you have done your due diligence and spent a little bit of time considering all of your available options.

Rob McNelly is a senior Power Systems solutions architect doing pre-sales and post-sales support for Meridian IT, headquartered in Deerfield, Illinois. McNelly was a technical editor for IBM Systems Magazine, and a former administrator within IBM’s Integrated Technology Delivery and Server Operations division. Prior to working for IBM, McNelly was an OS/400 and IBM i operator for many years for multiple companies. McNelly was named an IBM Champion for Power Systems in 2011, an IBM Champion Lifetime Achievement recipient in 2019, and can be reached at rob.mcnelly@gmail.com.

Edit: The first post for TechChannel in 2021.

Originally posted January 8, 2021

Moving forward, Rob McNelly will be writing regularly on AIX, IBM Power Systems hardware and related topics for TechChannel

“Hey, I heard you missed us, we’re back” – Van Halen, 1984

It’s a new year, and many changes are on the horizon. My AIXchange blog will remain on the website, and all of my previous work, going back to the early 2000s, is archived on robmcnelly.com—but moving forward, I’ll be writing regularly on AIX, IBM Power Systems hardware and related topics for TechChannel, producing new articles and passing on what I know. I’m also on Twitter (@robmcnelly).

In 2019 I received the IBM Power Champion Lifetime Achievement award. You may not realize it, but as part of that elite group, us lifetime achievers have massive pull. IBM doesn’t do anything without consulting us. So I recently drew up a list of things IBM should explore to grow the POWER community and skills in general.

Disclaimer: I have no pull and no one asked me to do this. Of course being an IBM Champion is a great honor, but I was just joking. What has actually happened is, based on many discussions with lots of smart people inside and outside of IBM, I’ve formed some ideas. Certainly some of these items may be more practical or feasible than others, and there may be perfectly valid reasons why certain things will never happen. But if the items on this list become reality, I believe our community would benefit. Feel free to circulate this information, and share your thoughts with me on this or any other topics you’d like to see me explore.

1) IBM should explicitly state that AIX can be used for non-commercial personal use. The reality is that we already have access to AIX in our everyday jobs, so why not be clear about what can be done with it? With or without official word, I imagine IBM lawyers have better things to do than go after people who play with AIX in their basements. Still, it’d be nice to see that written down—and if it already is, it should be more prominently stated.

I can take this further: Why not find cost-effective ways for hobbyists or students to access older POWER7 or POWER8 hardware as it comes off lease or reaches end of life? And what about a low-cost SAN option to connect everything? Throw in a virtual HMC image while we’re at it. Sure, I understand that this is no small amount of hardware to keep in one’s basement, but in an industry where so many are at or nearing retirement age, it seems short-sighted to not help people who want to spend their off-hours trying to get better at what they do. Let’s make it easier. Let’s dream a bit.

2) Provide free or very low-cost access to cloud instances. Like other tech giants, IBM is a cloud provider. Letting potential customers try out your interface and log into your systems seems like a good way to get further established in a highly competitive marketplace. Users could report bugs or usability issues to IBM, which would be better than having actual paying customers find and report them. Allowing greater access means greater mainstream familiarity with the product, which would come in handy when IBM pitches to clients that are considering cloud. If admins already know and like using the interface, that could spur more customers migrate to Power Systems.

And it wouldn’t take much CPU, memory and disk for IBM to make these instances available. Just limit individuals to a small percentage of a CPU, and many users could effectively share the same hardware. Imagine how much the ecosystem could grow if people could simply log into the systems and learn. Think how easy it is to download and use various Linux distributions. Shouldn’t it be at least that easy to run AIX on Power hardware? 

IBM does have its Cloud Free offering, but why not allow access to hobbyists and students without making them provide billing and payment information? The idea of clicking the wrong button fills me with dread.

Again, there may be valid reasons for this, but it’s my dream, and I’m dreaming big.

3) Make it easier to access the educational resources available through the IBM Academic Initiative. I get it: I can find a local school, partner with faculty, and gain access to the materials that way. Schools are also provided with system access for their students, the whole program is a fantastic idea. But the model of making me find a school first seems backwards. Why not provide me with the educational resources and systems in advance? That way, when I approach faculty at a local community college or high school, I’ll have a solid understanding of the materials, and they’ll have a good idea of the course content that I’m offering them.

4) Develop a low-cost, entry-level desktop system. To truly be proficient with Power Systems hardware, users would need, at the very least, access to and control of a virtual HMC and one system that could accommodate the creation of LPARs. Of course POWER9 desktop offerings are already available, but they’re fairly pricey and run Linux on Power (not AIX) out of the box. (Although I have seen AIX working via QEMU.)

But imagine if desktop hardware costs were more in line with what you’d spend on an Intel or AMD PC. Why is this important? Think about what made you a tech guru. Did you only study articles or Redbooks? I doubt it. Having hands-on access to systems allows AIX pros to practice critical tasks associated with operating system firmware, the HMC, VIOS, Live Partition Mobility, and much more. With fewer employers providing training or test boxes, people need opportunities to hone their skills and build confidence. You only need to run rm –rf * as root once to understand why you should never, ever do this. There’s simply no substitute for the learning experience that non-production environments provide.

Don’t get me wrong: I’m glad there’s a manufacturer selling fully open POWER9 hobbyist hardware, and I understand why that’s more expensive to build than other chipsets. We’re talking about massive differences in manufacturing scale. Still, I wish there was something more affordable. Either create the hardware within IBM or partner with another manufacturer, but do it. Just imagine what could be accomplished with something like this.

A final thing: Nigel Griffiths has a couple of new presentations. One is a POWER10 preview from the Power Virtual User Group (VUG) technical webinar series. Watch the replay and download the slides. Also watch the replay from his presentation on what’s new in AIX.