Logs and journalctl#

Concepts#

Why Logs Matter#

When something goes wrong on a Linux system (— )a service crashes, a login fails, a disk fills up, the gpu caught fire and is now burning your house down…) the answer is almost always in the logs. Logs record what happened, when, and often why.

Linux has two logging systems that work together:

  1. systemd journal (journalctl) — collects logs from all systemd services, the kernel, and system messages. Binary format, fast queries, structured data.
  2. Traditional log files (/var/log/) — text files written by rsyslog or syslog-ng. Older but still widely used.

/var/log/ — Traditional Log Files# 

ls /var/log/
File Content
syslog General system messages (Ubuntu)
messages General system messages (Debian, if configured)
auth.log Authentication events (logins, sudo, SSH)
kern.log Kernel messages
dpkg.log Package installation/removal history
apt/history.log APT command history
apt/term.log APT terminal output
boot.log Boot messages
dmesg Kernel ring buffer (hardware, drivers)
faillog Failed login attempts
lastlog Last login for each user
wtmp Login/logout history (binary, read with last) 
# View recent syslog entries
sudo tail -20 /var/log/syslog

# Follow syslog in real time
sudo tail -f /var/log/syslog

# Search for SSH events
sudo grep "sshd" /var/log/auth.log | tail -10

# View package install history
cat /var/log/dpkg.log | tail -20

Log Rotation#

Log files grow continuously. logrotate automatically compresses and rotates old logs:

ls /var/log/syslog*
# syslog         ← current
# syslog.1       ← yesterday
# syslog.2.gz    ← older, compressed
# syslog.3.gz
# ...

# Configuration
cat /etc/logrotate.conf
ls /etc/logrotate.d/

journalctl — The systemd Journal#

journalctl queries the systemd journal — a centralized, structured log that captures output from all services.

Basic Usage# 

# Show all logs (oldest first) — very long, use with caution
journalctl

# Show all logs, newest first (most useful)
journalctl -r

# Show the last 50 entries
journalctl -n 50

# Follow in real time (like tail -f)
journalctl -f

# No pager (output directly to terminal)
journalctl --no-pager -n 20

Filter by Unit (Service)# 

# Logs for a specific service
journalctl -u ssh
journalctl -u nginx
journalctl -u NetworkManager

# Follow a specific service's logs
journalctl -f -u ssh

# Last 20 entries for a service
journalctl -u ssh -n 20

Filter by Time# 

# Since a specific time
journalctl --since "2024-10-15 08:00:00"
journalctl --since "1 hour ago"
journalctl --since "today"
journalctl --since "yesterday"

# Time range
journalctl --since "2024-10-15" --until "2024-10-16"
journalctl --since "09:00" --until "10:00"

Filter by Priority#

Log priorities (syslog levels):

Priority Name Meaning
0 emerg System is unusable
1 alert Immediate action required
2 crit Critical conditions
3 err Error conditions
4 warning Warning conditions
5 notice Normal but significant
6 info Informational
7 debug Debug messages 
# Show only errors and above
journalctl -p err

# Show warnings and above
journalctl -p warning

# Show only critical
journalctl -p crit

Filter by Boot# 

# Current boot only
journalctl -b

# Previous boot
journalctl -b -1

# List recorded boots
journalctl --list-boots

Kernel Messages# 

# Kernel messages only (like dmesg)
journalctl -k

# Kernel messages from current boot
journalctl -k -b

Output Formats# 

# Short (default — looks like syslog)
journalctl -u ssh -n 5

# Verbose (all fields)
journalctl -u ssh -n 5 -o verbose

# JSON (for scripts)
journalctl -u ssh -n 5 -o json-pretty

# Only the message (no timestamps or metadata)
journalctl -u ssh -n 5 -o cat

Journal Storage and Size# 

# See how much disk space the journal uses
journalctl --disk-usage

# Reduce journal size
sudo journalctl --vacuum-size=500M    # keep only 500MB
sudo journalctl --vacuum-time=7d      # keep only last 7 days

# Configuration: /etc/systemd/journald.conf
# SystemMaxUse=500M    ← max disk usage
# MaxRetentionSec=1month  ← max age

By default on both Ubuntu and Debian, the journal is stored in /var/log/journal/ and persists across reboots. If /var/log/journal/ does not exist, logs are stored in /run/log/journal/ (RAM) and lost on reboot.

dmesg — Kernel Ring Buffer#

dmesg shows kernel messages, particularly useful for hardware issues:

dmesg                           # all kernel messages
dmesg | tail -20                # recent messages
dmesg -T                        # human-readable timestamps
dmesg -l err,warn               # only errors and warnings
sudo dmesg -w                   # follow (like tail -f for kernel)

# Common use: check for hardware issues after plugging in a device
dmesg -T | tail -20

Lab#

Exercise 1: Explore /var/log# 

# List log files
ls -lh /var/log/

# View recent system log entries
sudo tail -20 /var/log/syslog 2>/dev/null || sudo tail -20 /var/log/messages

# View auth log
sudo tail -10 /var/log/auth.log

# Check package history
tail -20 /var/log/dpkg.log

# View rotated logs
ls /var/log/syslog*

Exercise 2: Basic journalctl# 

# Recent logs (last 20 entries)
journalctl -n 20

# Newest first
journalctl -r -n 20

# Current boot only
journalctl -b -n 20

# Follow in real time (Ctrl+C to stop after watching for a moment)
journalctl -f &
sleep 3
# Generate a log entry
logger "Test log message from the course lab"
sleep 1
kill %1

Exercise 3: Filter by Service# 

# SSH logs
journalctl -u ssh -n 10

# List all units that have journal entries
journalctl --field _SYSTEMD_UNIT | head -20

# Follow a specific service
journalctl -f -u ssh &
sleep 3
kill %1

Exercise 4: Filter by Time and Priority# 

# Logs from the last hour
journalctl --since "1 hour ago" -n 20

# Logs from today
journalctl --since "today" -n 10

# Only errors and above
journalctl -p err -b

# Warnings from the last 24 hours
journalctl -p warning --since "24 hours ago" -n 20

Exercise 5: Kernel Messages# 

# View kernel messages
journalctl -k -n 20

# Or using dmesg
dmesg -T | tail -20

# Check for errors
dmesg -l err,warn | tail -10

Exercise 6: Journal Maintenance# 

# Check journal disk usage
journalctl --disk-usage

# List available boots
journalctl --list-boots

# See the journal configuration
cat /etc/systemd/journald.conf | grep -v "^#" | grep -v "^$"

Exercise 7: Using logger#

logger writes messages to the system log — useful in scripts:

# Write a test message
logger "Hello from the lab exercise"

# Check it appeared
journalctl -n 5
# You should see your message

# Write with a specific priority
logger -p user.warning "This is a warning message"
journalctl -p warning -n 5

# Write with a tag (identifier)
logger -t myapp "Application started"
journalctl -t myapp

Review#

1. What is the difference between `/var/log/` files and the systemd journal?

/var/log/ contains traditional text log files written by rsyslog/syslog-ng. The systemd journal is a binary, structured logging system accessed via journalctl. Both coexist — the journal captures everything from systemd services and the kernel, while /var/log/ files are more traditional and text-based.

2. How do you view logs for a specific service?

journalctl -u service-name. For example, journalctl -u ssh -n 20 shows the last 20 SSH log entries. Add -f to follow in real time.

3. How do you see only errors in the journal?

journalctl -p err shows messages with priority “error” and above (error, critical, alert, emergency).

4. How do you follow logs in real time?

journalctl -f (all logs) or journalctl -f -u service-name (specific service). For traditional files: sudo tail -f /var/log/syslog.

5. What does `journalctl -b` show?

Logs from the current boot only. journalctl -b -1 shows logs from the previous boot. journalctl --list-boots lists all recorded boots.

6. What is logrotate?

A tool that automatically rotates, compresses, and removes old log files to prevent them from filling the disk. It is configured in /etc/logrotate.conf and /etc/logrotate.d/.

7. What is `dmesg` and when is it most useful?

dmesg shows the kernel ring buffer — messages from the kernel about hardware, drivers, and kernel operations. It is most useful for diagnosing hardware issues, checking what happens when you plug in a device, or investigating boot problems.

Previous: systemd and Services | Next: Your First Script