Auditing System Events

To keep an operating system secure and reliable, you need to watch its operation closely. This includes monitoring, log analysis and, of course, auditing system events. Although auditing is not a means of protection against incidents by itself, it plays an important role in detecting security breaches and fixing them afterwards.

For this purpose there is a dedicated utility called auditd, which monitors system operations and helps analyze them. The audit system places triggers on the functions that handle system calls (the fundamental operations in the system kernel). These triggers then fire according to rules defined by the administrator. Information about all these actions is written to a log that can be analyzed later.

With auditd you can track a variety of events, such as system startup and shutdown, application execution, file access and permission changes, changes to user and group data, creation of network connections, changes to network settings, and much more.

Installation

In most cases, the audit package is already installed by default on CentOS systems.

On Debian / Ubuntu, you can install it with the following command:

sudo apt-get install auditd

The installed package includes several utilities:

• auditctl — controls the audit system: shows its current state, adds and deletes rules;
• autrace — audits events triggered by processes, similar to the strace utility;
• ausearch — searches for events in the logs;
• aureport — generates reports on audit activity.

The main configuration parameters of the auditd daemon (such as log format, flush frequency, maximum size, and others) are set in the configuration file /etc/audit/auditd.conf. The default settings are usually sufficient for auditing, but you can make additional changes as needed. Detailed descriptions of the parameters can be found, for example, here, as well as in the manual (man auditd.conf).

Creating rules

Rules are configured with the auditctl utility and the following options:

• -l — list the existing rules (right after auditd is installed, the list is empty);
• -a — add a new rule;
• -d — delete a rule from the list;
• -D — delete all rules.

New rules are created with the following command:

sudo auditctl -a list,action -S syscall_name -F filter

The list is the set of events the rule applies to. You can think of the list as a filter that makes the rule more precise.

There are five kinds of lists:

• task — events related to process creation;
• entry — events that occur on entry to a system call;
• exit — events that occur when a system call finishes;
• user — events related to user space parameters (uid, pid, and gid);
• exclude — used to exclude events from tracking.

In practice, entry and exit are used most often.

The action defines what to do when the event occurs: write it to the log (always) or not (never).

The system call name specifies which call should fire the trigger and capture the event (for example, open, close, exit, and others).

The filter is an optional setting used to add extra parameters. For example, to track access to files in the /etc directory, you can specify it here:

sudo auditctl -a exit,always -S open -F path=/etc/

You can use several filters to narrow down the events you are interested in, such as file changes with the w (write) and a (attribute change) permissions:

sudo auditctl -a exit,always -S open -F path=/etc/ -F perm=wa

When setting up file tracking rules, you can omit the system call name (the -S option):

sudo auditctl -a exit,always -F path=/etc/ -F perm=wa

To watch a specific file, you can use an even shorter form (the -p option replaces perm):

sudo auditctl -w /etc/passwd -p wa

The audit.rules file

Rules can also be defined from the console and saved to the /etc/audit/audit.rules file. In this case they are applied permanently.

The rule syntax is the same, you just omit the auditctl command, for example:

-w /etc/passwd -p wa

The beginning of the file usually contains meta rules that set basic parameters:

# Clear all previous rules
-D 

# Set the number of buffers for storing messages 
-b 320 

# Action on buffer overflow: 0 — do nothing; 1 — send a message to dmesg, 2 — trigger a kernel panic 
-f 1

User rules follow after that.

Below are examples of rules for monitoring various events:

# Watch the audit system configuration files 
-w /etc/audit/auditd.conf -p wa 
-w /etc/audit/audit.rules -p wa

# Watch the log files 
-w /var/log/audit/ 
-w /var/log/audit/audit.log 

# Watch at settings and jobs 
-w /var/spool/at 
-w /etc/at.allow 
-w /etc/at.deny 

# Monitor password and group files 
-w /etc/group -p wa 
-w /etc/passwd -p wa 
-w /etc/shadow 

# Watch login configuration and log files 
-w /etc/login.defs -p wa 
-w /etc/securetty 
-w /var/log/faillog 
-w /var/log/lastlog 

# Monitor the host list and host names 
-w /etc/hosts -p wa 

# Watch daemon startup scripts 
-w /etc/init.d/ 
-w /etc/init.d/auditd -p wa 

# Watch SSH server settings 
-w /etc/ssh/sshd_config 

# Track file permission changes 
-a entry,always -S chmod -S fchmod -S chown -S chown32 -S fchown -S fchown32 -S lchown -S lchown32 

# Track file creation, opening, and resizing 
-a entry,always -S creat -S open -S truncate -S truncate64 -S ftruncate -S ftruncate64 

# Track directory creation and deletion 
-a entry,always -S mkdir -S rmdir 

# Track link deletion and creation 
-a entry,always -S unlink -S rename -S link -S symlink 

# Monitor file system mount operations 
-a entry,always -S mount -S umount -S umount2

After changing the configuration file, restart auditd to apply the changes:

sudo service auditd restart

Analyzing audit logs

The audit system log files are stored in /var/log/audit. They are analyzed with the ausearch and aureport tools, which offer extensive functionality and let you build informative reports from log data based on the criteria you specify.

Additional parameters and options for these tools can be found in the documentation (man aureport and man ausearch) or online, for example, here.

To generate a report on files, use aureport with the -f option:

sudo aureport -f

You can also limit the report to a specific time interval:

sudo aureport --start month/day/year hours:minutes:seconds --end month/day/year hours:minutes:seconds

# For example:

sudo aureport -f --start 08/20/20 12:00 --end 08/20/20 13:00

You can also use the following keywords to specify time intervals:

• now (the current time),
• recent (the last ten minutes),
• today (today, starting from midnight),
• yesterday (yesterday),
• this-week (the current week),
• this-month (the current month),
• this-year (the current year).

The command output contains a lot of information. To make analysis easier, use the --summary option to get a brief overview of accesses to each file.

sudo aureport -f -i --start recent --summary

If you notice a suspicious access attempt, you can identify the process that triggered the event:

sudo aureport -f -i --start today | grep /etc/passwd

Then each event can be analyzed in more detail with the ausearch tool:

sudo ausearch -a event_number

Ausearch can also find events based on various parameters, such as:

User ID (with the -ui option):

sudo ausearch -ui 1111 --interpret

(The --interpret or -i option displays the data in a human-readable format.)

Executable name (with the -x option):

sudo ausearch -x /path/to/file

Daemon name (with the -tm option):

sudo ausearch -tm cron

System call (with the -sc option):

sudo ausearch -sc ptrace

To limit the results by time, use the same syntax as for aureport.

The full list of available ausearch options can be found in the documentation (man ausearch) or online, for example, here.