4.6. Red Hat Linux-Specific Information

to start with the more straightforward tools, and work from there.

Using free, it is possible to get a concise (if somewhat simplistic) overview of memory and swap utilization. Here is an example:

    522104          0     522104
      

Contrast that example with this one:

    530136     111308     418828
      

This system has about 256MB of RAM, the majority of which is in use, leaving only about 8MB free. Over 100MB of the 512MB swap partition is in use. Although this system is certainly more limited in terms of memory than the first system, to see if this memory limitation is causing performance problems we must dig a bit deeper.

Although more cryptic than free, vmstat has the benefit of displaying more than memory utilization statistics. Here is the output from vmstat 1 10:

 0   0     6    10  120    24  10   2  89
 2  0  0 111304   9728   7036 107204   0   0     0     0  526  1653  96   4   0
 1  0  0 111304   9616   7036 107204   0   0     0     0  552  2219  94   5   1
 1  0  0 111304   9616   7036 107204   0   0     0     0  624   699  98   2   0
 2  0  0 111304   9616   7052 107204   0   0     0    48  603  1466  95   5   0
 3  0  0 111304   9620   7052 107204   0   0     0     0  768   932  90   4   6
 3  0  0 111304   9440   7076 107360  92   0   244     0  820  1230  85   9   6
 2  0  0 111304   9276   7076 107368   0   0     0     0  832  1060  87   6   7
 3  0  0 111304   9624   7092 107372   0   0    16     0  813  1655  93   5   2
 2  0  2 111304   9624   7108 107372   0   0     0   972 1189  1165  68   9  23
      

During this 10-second sample, the amount of free memory (the free field) varies somewhat, and there is a bit of swap-related I/O (the si and so fields), but overall this system is running well. It is doubtful, however, how much additional workload it could handle, given the current memory utilization.

When researching memory-related issues, it is often necessary to see how the Red Hat Linux virtual memory subsystem is making use of system memory. By using sar, it is possible to look at this aspect of system performance in much more detail.

By reviewing the sar -r report, we can look more closely at memory and swap utilization.:

Linux 2.4.18-18.8.0smp (raptor.example.com)     12/16/2002

12:00:01 AM kbmemfree kbmemused  %memused kbmemshrd kbbuffers  kbcached
12:10:00 AM    240468   1048252     81.34         0    133724    485772
12:20:00 AM    240508   1048212     81.34         0    134172    485600
…
08:40:00 PM    934132    354588     27.51         0     26080    185364
Average:       324346    964374     74.83         0     96072    467559
      

The kbmemfree and kbmemused fields show the typical free and used memory statistics, with the percentage of memory used displayed in the %memused field. The kbbuffers and kbcached fields show how many kilobytes of memory are allocated to buffers and the system-wide data cache.

The kbmemshrd field is always zero for systems (such as Red Hat Linux) that use the 2.4 Linux kernel.

The lines for this report have been truncated to fit on the page. Here is the remainder of each line, with the timestamp added to the left to make reading easier:

12:00:01 AM   kbswpfree kbswpused  %swpused
12:10:00 AM      522104         0      0.00
12:20:00 AM      522104         0      0.00
…
08:40:00 PM      522104         0      0.00
Average:         522104         0      0.00
      

For swap utilization, the kbswpfree and kbswpused fields show the amount of free and used swap space, in kilobytes, with the %swpused field showing the swap space used as a percentage.

To learn more about the swapping activity is taking place, use the sar -W report. Here is an example:

Linux 2.4.18-18.8.0 (pigdog.example.com)     12/17/2002

12:00:01 AM  pswpin/s pswpout/s
12:10:01 AM      0.15      2.56
12:20:00 AM      0.00      0.00
…
03:30:01 PM      0.42      2.56
Average:         0.11      0.37
      

Here we can see that, on average, there were three times fewer pages being brought in from swap (pswpin/s) as there were going out to swap (pswpout/s).

To better understand how pages are being used, refer to the sar -B report:

Linux 2.4.18-18.8.0smp (raptor.example.com)     12/16/2002

  30336     49275
…
08:40:00 PM      0.00      7.79     71236      1371      6760     15873
Average:       201.54    201.54    169367     18999     35146     44702
      

Here we can view how many blocks per second are paged in from disk (pgpgin/s) and paged out to disk (pgpgout/s). These statistics serve as a barometer of overall virtual memory activity.

However, more knowledge can be gained by looking at the other fields in this report. The Red Hat Linux kernel marks all pages as either active or inactive. As the names imply, active pages are currently in use in some manner (as process or buffer pages, for example), while inactive pages are not. This example report shows that the list of active pages (the activepg field) averages approximately 660MB[1].

The remainder of the fields in this report concentrate on the inactive list — pages that, for one reason or another, have not recently been used. The inadtypg field shows how many inactive pages are dirty (modified) and may need to be written to disk. The inaclnpg field, on the other hand, shows how many inactive pages are clean (unmodified) and do not need to be written to disk.

The inatarpg pool for page replacement purposes.

For additional insight into page status (specifically, how often pages change status), use the sar -R report. Here is a sample report:

Linux 2.4.18-18.8.0smp (raptor.example.com)     12/16/2002

12:00:01 AM   frmpg/s   shmpg/s   bufpg/s   campg/s
12:10:00 AM     -0.10      0.00      0.12     -0.07
12:20:00 AM      0.02      0.00      0.19     -0.07
…
08:50:01 PM     -3.19      0.00      0.46      0.81
Average:         0.01      0.00     -0.00     -0.00
      

The statistics in this particular sar report are unique, in that they may be positive, negative, or zero. When positive, the value indicates the rate at which pages of this type are increasing. When negative, the value indicates the rate at which pages of this type are decreasing. A value of zero indicates that pages of this type are neither increasing or decreasing.

In this example, the last sample shows slightly over three pages per second being allocated from the list of free pages (the frmpg/s field) and nearly 1 page per second added to the page cache (the campg/s field). The list of pages used as buffers (the bugpg/s field) gained approximately one page every two seconds, while the shared memory page list (the shmpg/s field) neither gained nor lost any pages.