Analyzing thread CPU usage on Linux

Use top to analyze the CPU usage of each application thread.

Introduction

To investigate the per-thread CPU usage on Linux, the recommended tool is top with the -H option, which provides an additional thread for information not provided by top in the default usage.

Understanding the top -H output

The output of top -H on Linux shows the breakdown of the CPU usage on the machine by individual threads. The top output has the following sections of interest:

 
  top - 16:15:45 up 21 days,  2:27,  3 users,  load   average: 17.94, 12.30, 5.52   
  Tasks: 150 total,  26 running, 124 sleeping,   0   stopped,   0 zombie   
  Cpu(s): 87.3% us,  1.2% sy,  0.0% ni, 27.6% id,  0.0%   wa,  0.0% hi,  0.0% si   
  Mem:   4039848k total,  3999776k used,   40072k free,    92824k buffers   
  Swap:  2097144k total,      224k used, 2096920k free,  1131652k cached

The Cpus(s) row in this header section shows the CPU usage in terms of the following:

us: Percentage of CPU time spent in user space.
sy: Percentage of CPU time spent in kernel space.
ni: Percentage of CPU time spent on low priority processes.
id: Percentage of CPU time spent idle.
wa: Percentage of CPU time spent in wait (on disk).
hi: Percentage of CPU time spent handling hardware interrupts.
si: Percentage of CPU time spent handling software interrupts.

The "us", "sy" and "id" values are useful as the user, system (kernel) and idle CPU time respectively.

The next section shows the per-thread breakdown of the CPU usage.

 
    PID USER    PR  NI  VIRT  RES  SHR S %CPU   %MEM    TIME+  COMMAND   
  31253 user1   16   0 2112m 2.1g 1764 R 37.0   53.2   0:39.89 java   
  31249 user1   16   0 2112m 2.1g 1764 R 15.5   53.2   0:38.29 java   
  31244 user1   16   0 2112m 2.1g 1764 R 13.6   53.2   0:40.05 java   
  31250 user1   16   0 2112m 2.1g 1764 R 13.6   53.2   0:41.23 java   
  31242 user1   16   0 2112m 2.1g 1764 R 12.9   53.2   0:40.56 java   
  31238 user1   16   0 2112m 2.1g 1764 S 12.6   53.2   1:22.21 java   
  31246 user1   16   0 2112m 2.1g 1764 R 12.6   53.2   0:39.62 java   
  31248 user1   16   0 2112m 2.1g 1764 R 12.6   53.2   0:39.40 java   
  31258 user1   16   0 2112m 2.1g 1764 R 12.6   53.2   0:39.98 java   
  31264 user1   17   0 2112m 2.1g 1764 R 12.6   53.2   0:39.54 java   
  31243 user1   16   0 2112m 2.1g 1764 R 12.2   53.2   0:37.43 java   
  31245 user1   16   0 2112m 2.1g 1764 R 12.2   53.2   0:37.53 java   
  31255 user1   16   0 2112m 2.1g 1764 R 11.9   53.2   0:40.84 java   
  31265 user1   16   0 2112m 2.1g 1764 R 11.9   53.2   0:40.46 java   
  31239 user1   16   0 2112m 2.1g 1764 S 11.6   53.2   1:22.79 java   
  31254 user1   16   0 2112m 2.1g 1764 R 11.6   53.2   0:41.12 java   
  31266 user1   16   0 2112m 2.1g 1764 R 11.2   53.2   0:40.47 java   
  31261 user1   16   0 2112m 2.1g 1764 R 10.9   53.2   0:39.31 java   
  31262 user1   16   0 2112m 2.1g 1764 R 10.9   53.2   0:38.50 java   
  ..

This provides the following information, some of which is for the whole process, and other inforamation relevant to the particular thread:

PID

The thread ID. This can be converted into hexadecimal and used to correlate to the "native ID" in a javacore.txt file.

USER

The user ID of the user that started the process.

PR

The priority of the thread.

NI

The "nice" value for the process.

VIRT

The virtual memory (allocated) usage of the process.

RES

The resident memory (committed) usage of the process.

SHR

The shared memory usage of the process.

S

The state of the thread. This can be one of the following:

R: Running
S: Sleeping
D: Uninterruptible sleep
T: Traced
Z: Zombie

%CPU

The percentage of a single CPU usage by the thread.

%MEM

The percentage of the memory used by the process.

TIME+

The amount of CPU time used by the thread.

COMMAND

The name of the process executable.

Note that the "Cpu(s)" line in the header of the output shows the percentage usage across all of the available CPUs, whereas the %CPU column represents the percentage usage of a single CPU. For example, on a four-CPU machine the Cpu(s) row will total 100% and the %CPU column will total 400%.

What to look for from the top -H output

In the per-thread breakdown of the CPU usage shown above, the Java process is taking approximately 75% of the CPU usage. This value is found by totalling the %CPU column for all the Java threads (not all threads are shown above) and dividing by the number of CPUs. The Java process is not limited by other processes, because the CPU there is still approximately 25% idle. You can also see that the CPU usage of the Java process is spread reasonably evenly over all of the threads in the Java process. This spread implies that no one thread has a particular problem. Although the application is allowed to use most of the available CPU, approximately 25% of the total CPU is idle meaning that some points of contention or delay in the Java process can be identified. A report indicating that active processes are using a small percentage of CPU, even though the machine appears idle, means that the performance of the application is probably limited by points of contention or process delay, preventing the application from scaling to use all of the available CPU. If a deadlock is present, the reported CPU usage for the Java process is low or zero. If threads are looping, the Java CPU usage approaches 100%, but a small number of the threads account for all of that CPU time. Where you have threads of interest, note the PID values because you can be convert them to a hexadecimal value and look up the threads in the javacore.txt file to discover if the thread is part of a thread pool. In this way you gain an understanding of the kind of work that the thread does from the thread stack trace in the javacore.txt file. For example, the PID 31253 becomes 7A15 in hexadecimal. This value maps to the "native ID" value in the javacore.txt file.

© Copyright IBM Corporation 2005, 2008. All Rights Reserved.
© Copyright Sun Microsystems, Inc. 1997, 2007, 901 San Antonio Rd., Palo Alto, CA 94303 USA. All rights reserved.
US Government Users Restricted Rights - Use, duplication or disclosure restricted by GSA ADP Schedule Contract with IBM Corp.
This information center is powered by Eclipse technology. (http://www.eclipse.org/)