Monitoring Load and Processes: top, htop, atop

Every active process on a server generates load, and when the server runs out of resources, this can cause various problems, such as a slow website or delays in script execution.

There are several dedicated utilities for analyzing server load. We will look at three of them:

• top
• htop
• atop

In general, they are quite similar. The htop utility provides an interactive interface and can be more convenient to use. Top, on the other hand, ships preinstalled with Linux distributions and does not need to be installed separately. Atop can keep logs.

If your website is slow, we also recommend using Google's PageSpeed Insights service. This tool analyzes page load speed and gives recommendations for optimizing your site. You can also improve site performance by moving the MySQL temporary files directory to tmpfs.

top

The top utility is installed by default on most UNIX systems. You can run it by connecting to the server via SSH and executing the top command.

The top output looks like this:

The upper part of the output shows system information, and below it is the list of processes. The output refreshes every 2 seconds, and the most resource-intensive processes appear at the top of the list.

System Information

The upper-left corner of the output shows the current system time, followed by this data:

Up — the time elapsed since the last system reboot.

User — the number of current users.

Load average — the average server load over the last one, five, and fifteen minutes.

Tasks — the total number of processes in various states, such as running, sleeping, stopped, and zombie (child processes waiting for the parent process to finish).

Cpu(s) — the percentage of CPU time spent on processes, broken down into the following categories:

us — user processes (a high value may indicate problems in the site code that need optimization).

sy — system processes.

id — idle resources (the higher this value, the better).

wa — I/O operations, i.e., disk operations.

Mem, Swap — RAM usage information, including total (total amount), free (free amount), and used (used amount).

Process Information

By default, processes are shown as a table sorted by CPU usage, from highest to lowest.

The columns are described below:

PID — process ID;

USER — the user who started the process;

PR — process priority;

NI — adjusted priority (set by the user with the nice command);

VIRT — virtual memory size (the amount of memory requested by the process, even if it actually uses less);

RES — the amount of physical RAM used by the process (if a process requested 50 MB of memory but uses only 10 MB, 10 MB is shown);

SHR — the amount of memory shared with other processes (memory that other processes can use);

S — process status (running — running; sleeping — waiting; zombie — zombie process);

%CPU — percentage of CPU time used;

%MEM — percentage of RAM used;

TIME — total running time of the process;

COMMAND — process name (the command the process was started with).

Controls

The following keys are used to work with top:

Space — refresh the output

M — sort by memory usage

P — sort by CPU load (default)

T — sort by process running time

A — sort by maximum consumption of various resources

u — filter by user name (you need to enter the user name)

k — kill a process (you need to specify its ID, the PID)

n — change the number of processes shown (you need to specify the desired number)

c — show the full path of the running process (COMMAND column)

h — show help

q — quit the program

htop

Unlike top, htop has to be installed on the server first:

Ubuntu / Debian:

apt-get install htop

CentOS:

yum install htop

After installation, run:

htop

The command output is shown below:

As with top, the upper part shows system information and the lower part shows the list of processes. The columns in htop are the same as in top (described above).

In the upper left, you can see the load on each CPU core, memory usage, the number of processes, the load average over the last 1, 5, and 15 minutes, and the system uptime.

By default, processes are sorted by CPU usage, from highest to lowest.

To sort by memory usage (or another parameter), simply click the name of the corresponding column, for example MEM. To reverse the sort order (from lowest to highest), click the same column again. You can also use the M (sort by memory), P (by CPU), and T (by time) keys, just like in top.

Additional functions include:

Space — tag a process (this lets you mark processes to perform group operations on them, such as killing them).

u — show only the processes of a specific user.

The F1–F10 keys are available for control:

F1 — show help.

F2 — display setup (adding or removing columns, showing additional information in the header, and other settings).

F3 — search for processes.

F4 — filter processes (show only processes with specific text in their names).

F5 — show the process tree (shows the relationships between parent and child processes).

F6 — change the sort order.

F7 / F8 — raise or lower process priority.

F9 — kill a process (unlike top, you do not need to specify the PID — just select the process and press F9; press Enter to confirm or Esc to cancel).

F10 — quit the program.

The function of each key may change depending on which program menu you are in. The available actions are shown at the bottom of the window, which makes the utility easier to use.

atop

The main feature of atop is its ability to keep logs. This lets you not only monitor the current load but also review how processes behaved on previous days, which helps identify intermittent errors that are hard to catch in real time.

To use the utility, install it on the server:

Ubuntu / Debian:

apt-get install atop -y

CentOS:

yum install atop -y

We also recommend enabling atop at startup:

Ubuntu / Debian / CentOS 7:

systemctl enable atop

CentOS 6:

chkconfig atop on

Ubuntu 14.04:

rm /etc/init/atop.override

Run the utility:

atop

The output will look roughly like this:

The upper part of the screen shows system information and the load on key components: CPU, cores, memory, and network. Below is the list of processes.

You can use the following keys to control the output:

m — sort by memory usage

d — sort by disk load

u — load per user

v — detailed process information

i — change the data refresh interval (10 seconds by default)

g — return to the default output

n — sort processes by network load (available with the kernel patch installed)

Key combinations with Shift sort the current process list by the corresponding parameter:

• Shift + m — sort by memory usage
• Shift + c — sort by CPU consumption (default)
• Shift + d — sort by disk usage
• Shift + n — sort by network usage

atop Logs

By default, atop records system state information every 10 minutes and saves it to a log file in the /var/log/atop directory.

To view the log for the current day, run:

atop -r

Useful key combinations:

• t — move forward in time
• Shift + t — move backward in time

The log file for a specific day is named atop_YYYYMMDD. To view the log for a specific day, use the atop -r command and specify the file path, for example:

atop -r /var/log/atop/atop_20200227

To change the logging settings, edit the atop configuration file located at /etc/default/atop or /etc/sysconfig/atop (on CentOS).

# Interval between server load snapshots, in seconds:
INTERVAL=600

# Path to the log directory:
LOGPATH="/var/log/atop"

# Log file name
OUTFILE="$LOGPATH/daily.log"

For example, to record load snapshots once a minute, set the interval to 60.

After making changes, restart atop:

systemctl restart atop.service