Edit: Did you know this tool exists? Some links no longer work.
Originally posted October 4, 2016 on AIXchange
I often hear from people who want to know how to conduct in-depth performance analysis on Linux. These folks are new to the platform and wonder why they can’t find many of the tools (PerfPMR, for instance) that they take for granted with AIX.
If you find yourself in this situation, you should know about a performance-focused script called the Linux Performance Customer Profiler Utility (lpcpu), which gathers data from both x86- and Power Linux-based systems:
“This script captures a lot of potentially interesting performance data (profile information, system information, and some system configuration information) on a single pass approach. Gathering all of this information at once allows the context to be understood for the performance profiling data to be analyzed.
- The script will check to be sure you have the basic tools installed on your system.
This script takes advantage of all of the “normal” performance tools used on Linux.
- iostat
- mpstat
- vmstat
- perf
- meminfo
- top
- sar
- oprofile
- perf
In addition, relevant system information is gathered, with the profiler output, into a single tarball saved on your system. By default, the file is saved in /tmp.
The script is designed to run on both x86 and Power servers, the focus being SLES and RHEL distros. It should work on OpenSUSE and Fedora as well.
The script creates a zipped tar-ball placed in /tmp by default. You can un-zip the file and poke around the data files captured to learn more about your system. Typically, the zipped tar-ball is returned to performance analysts at IBM who can help with problem determination and ideas.
In 95% of our interactions with product teams and customers, there is generally something easy to address first. There are naturally many other in-depth tools we might leverage in subsequent data runs, but first we want to be sure everyone is “on the same page.”
Here’s more on testing the script and processing the results:
“This checks for errors and attempts to run all of the default profilers. A typical error will be that the profiler being run is not installed. Obviously, in that case the profiler should be installed, or if not available, you can override the profiler list to skip that tool (but keep in mind that the data gathered may not be as useful).
You do need a number of rpm packages installed on your system.
- sysstat
- profile (this rpm is on the SDK image for SLES)
The script does assume the Linux kernel with symbol information is available. This depends on your distro and version since they are generally packaged differently. The script does parse and check all of the common correct places to find vmlinux (the unstripped kernel). On RHEL 6.2, you will need the kernel-debuginfo*.rpm packages installed.
This script is not targeted or focused on Java applications, but it does serve well as the first pass data gathering tool.
Typically, the workload being tested reaches a fairly steady state (has settled down) and performance data can be collected from the system.
In this case, you can run the script for the default 2 minutes, and the script will profile the information and gather everything together.
Along with the script, we have the ability to format the results into a series of html pages and charts.
Previous releases of LPCPU have required an x86 system for producing charts, however the latest release removes this requirement, unless you would prefer to force the old behavior to be used (see the README for details if you would like to force the old behavior).
Take the lpcpu.tar.bz2 file, and unpack it.
# cd /tmp
# tar -jxf lpcpu.tar.bz2
Copy the data tarball to the system you would like to host your data on (this could be the test system or a workstation). Unpack the tarball and issues the following commands:
# pwd
/var/www/html
# tar -jxf lpcpu_data.mysystem.default.2012-04-04_1147.tar.bz2
# cd lpcpu_data.mysystem.default.2012-04-04_1147/
# ./postprocess.sh /tmp/lpcpu/
<lots of messages>
# ls -l summary.html
View that summary.html file in a browser. Depending on your browser, you may need a web server to make use of the new charting abilities. A Python script is included for running a simple web server should you not have one available. If you cannot run the Python script and do not have a web server available, please fall back on the old charting method (see the README for details).”
If you’ve tried out the tool, I’d like to hear from you. What other things would you like to see it do in your environment?