System Logging with systemd

The systemd system manager is used in most modern Linux distributions, including Ubuntu, Debian and CentOS. It manages the server and services (for example, starting and stopping them) and is also responsible for logging. The systemd log, known as the journal, collects system messages from the kernel, services and applications. Logging with systemd can run alongside syslog or replace it completely.

The journalctl utility is used to work with the journal.

Viewing and Filtering Logs

Time Display in Logs

Journal entries can be viewed in local time or in UTC by adding the --utc option to the command.

By default, logs are displayed in local time. You can make sure the time zone is set correctly by running the following command:

timedatectl status

The current time zone is shown in the Local time line.

If you need to change it, you can view the list of available time zones:

timedatectl list-timezones

Then set the time zone you want:

timedatectl set-timezone timezone

After that, check the settings again with the timedatectl status command.

Filtering by a Specific Boot

To show events from the last (current) boot, use the -b option:

journalctl -b

If you need to display entries with timestamps in UTC:

journalctl -b --utc

To list the available boots:

journalctl --list-boots

Sample output:

-2 b94zf405f9424a1d39b667ce0bdbcff7 Wed 2020-07-08 08:34:12 MSK—Wed 2020-07-08 08:41:22 MSK
-1 t80dc883d180dc04z5db0abcb0dc5fa2 Wed 2020-07-08 09:01:46 MSK—Wed 2020-07-08 09:28:08 MSK
0 8cabc9420ea34a8d868e3014895269d8 Wed 2020-07-08 11:56:57 MSK—Wed 2020-07-08 13:51:27 MSK

To view the data for a specific boot from the list, specify its sequence number (the first column of the output) or its ID (the second column):

journalctl -b -1

# Or:

journalctl -b t80dc883d180dc04z5db0abcb0dc5fa2

If previous boots are not saved by default, you need to change the configuration file:

sudo nano /etc/systemd/journald.conf

Then set the Storage parameter to persistent:

[Journal]
Storage=persistent

Filtering by Time Range

If you are interested in a period that does not match previous boots, you can specify a particular time window. The following options are used for this:

• --since (starting from)
• --until (up to)
• The time is specified in the format: "YYYY-MM-DD HH:MM:SS"
• You can also use the keywords yesterday, today and now

For example, to view events from a specified time up to the current moment:

journalctl --since "2020-07-06 07:00:00"

To show events for a specific period of time:

journalctl --since "2020-07-06 07:00:00" --until "2020-07-06 08:00:00"

And to see events since the previous day, use:

journalctl --since yesterday

Filtering by Service (Unit)

If you are only interested in the events of a specific service, you can use the command:

journalctl -u service_name

You can add time parameters if needed. For example, to view Nginx events for certain hours, run:

journalctl -u nginx.service --since 10:00 --until 16:00

You can also specify several units:

journalctl -u nginx.service -u php-fpm.service --since today

Filtering Kernel Events

You can show only kernel events using the -k option:

journalctl -k -2

Filtering by User, Group or Process

You can also view data for a specific PID, UID or GID:

journalctl _PID=process_id
journalctl _UID=user_id 
journalctl _GID=group_id

For example, to view only events related to a specific process for the current day, run the following command:

journalctl _PID=544 --since today

To get a list of all IDs that have entries in the journal, use:

journalctl -F _UID 

# Similarly for GID:

journalctl -F _GID

Filtering by Priority

With the -p option, you can show events starting from a certain priority level. The levels are:

• 0: emerg (emergency, the system is unusable)
• 1: alert (immediate action required)
• 2: crit (critical condition)
• 3: err (error)
• 4: warning (warning)
• 5: notice (notice)
• 6: info (informational)
• 7: debug (debugging)

In the command, you can specify either the name of the level or its number.

For example, the following command shows all events in the current boot with level 3 (error) and higher:

journalctl -p err -b 

# Or:

journalctl -p 3 -b

Display Settings

Basic Settings

Truncating long lines to fit the terminal window:

journalctl --no-full

Printing entries to standard output (this lets you process entries with other utilities, such as grep, or save them to a text file):

journalctl --no-pager

Output Formats

You can specify the output format with -o:

journalctl -o format

Available formats:

• cat — only the text of log messages, without additional information;
• export — a binary format suitable for exporting or backing up journals;
• json — standard .json format with one entry per line;
• json-pretty or json-sse — human-readable .json format;
• short — an output format similar to syslog;
• short-iso — syslog output format with ISO 8601 timestamps;
• short-monotonic — syslog output format with monotonic timestamps;
• short-precise — syslog output format with time down to microseconds;
• verbose — the most detailed output with additional information.

Example:

journalctl -b -u nginx.service -o json-pretty

Showing Recent Events

By default, the last 10 events are shown:

journalctl -n

You can specify the number of events, for example:

journalctl -n 20

Showing events in real time:

journalctl -f

Managing the Journal

Disk Usage

To find out how much disk space the journal takes up, run the command:

journalctl --disk-usage

Limiting the Journal Size

You can set journal size limits by editing its configuration file:

sudo nano /etc/systemd/journald.conf

You can configure the following parameters:

• SystemMaxUse= The maximum amount of disk space the journal can take up.
• SystemKeepFree= The amount of disk space that must remain free after logs are saved.
• SystemMaxFileSize= The maximum size of a journal file; once it is reached, the file is rotated.
• RuntimeMaxUse= The maximum amount of space logs can take up in the /run file system.
• RuntimeKeepFree= The amount of space that must remain free in the /run file system after logs are saved.
• RuntimeMaxFileSize= The maximum size of a journal file in the /run file system; once it is reached, the file is rotated.

Deleting Old Entries

There are two ways to clear old entries and reduce the size of the journal.

1. You can specify the size to which the journal should be reduced. Old entries will be deleted until the journal reaches the specified size. For example:

sudo journalctl --vacuum-size=1G

2. You can specify the period of time for which data should be kept. In this case, all entries older than the specified period will be deleted. For example:

sudo journalctl --vacuum-time=1year

This will delete all entries except those created within the last year.