What is Crush in AIX, Anyway?

Edit: This link sheds a little more light on the crush command. I still think this intro holds up.

Originally posted August 5, 2008 on AIXchange

Are administrators violent by nature? Given our terminology, I sometimes wonder.

Consider: When a machine fails, it crashes. If a user has a process that doesn’t behave, we kill it. When we want to manipulate data, we use cut–or in perl, we use chop. We can even use finger if it’s appropriate. I guess that’s not as bad as “the” finger.

Of course, we do have our soft side. Nice, cat, and sleep are other commands we use. So maybe it just depends on our mood.

AIX also has a command called crush. Have you used it? If you have bos.perf.tools installed, you can find it in:

/usr/lib/perf/crush

When I googled /usr/lib/perf/crush, it returned one page, in Chinese. A Google translation indicated that it was a listing of files that are contained in the bos.perf.tools fileset.

I asked around, but I couldn’t find any other information. So I got on a test box and I ran crush. The machine returned this output:

Please supply an integer number of pages.

So I did. I ran /usr/lib/perf/crush 1. Nothing exciting seemed to happen.

Then someone told me that it was an undocumented command. Another guy told me it’s undocumented for a reason. A third guy told me that all crush does is allocate a bunch of memory. Then it goes through and touches each page in memory and cleans up the cruft that accumulates in the machine while it’s been running.

What you will find, when you give it a large enough integer number of pages, is that your free list will grow, and it will page some of your memory out to paging space.

In my case, I ran:

svmon -G
                     size          inuse         free         pin         virtual
memory       958464     957272       1192     216590     520248
pg space     786432     245608

                 work         pers         clnt
pin            216590          0             0
in use       331024           0       626248

I took the memory size (958464), subtracted my pinned memory (216590) and tried the result (roughly 740000):

/usr/lib/perf/crush 740000

When I reran svmon, I saw:

svmon -G
                    size           inuse         free           pin          virtual
memory       958464     248584     709880     216532     520256
pg space     786432     271874

                work         pers       clnt
pin           216532           0           0
in use       246062          0      2522

My free list had gone from 1192 to 709880.  After a few hours, I saw:

svmon -G
                    size           inuse           free         pin          virtual
memory       958464     548408     410056     216596     517442
pg space     786432     265683

                 work       pers       clnt
pin           216596          0            0
in use       289660          0      258748

With this post, at least the next person who searches for /usr/lib/perf/crush will have a little more to go on. Hopefully whoever wrote crush will see this and leave a comment so we can better understand when it’s appropriate to run the tool on a production machine, and what it’s meant for in the first place.