systemd & journalctl

What it is and where it fits 🎯#

systemd is the init system and service manager on essentially every modern Linux distribution — PID 1, the first process the kernel starts, responsible for bringing up every other service in dependency order and supervising them afterward. systemctl is the control surface (start/stop/enable/status a unit); journalctl reads the structured, binary journal systemd's logging component (journald) writes to, replacing the older plain-text /var/log/*.log + syslog model on most distros. These are the first two tools for "why did this service stop / why won't it start" — everything else in Linux troubleshooting (network inspection, process tracing) usually comes after confirming whether the service is even running.

A unit's lifecycle, and where each command touches it#

Service status and control#

systemctl status nginx
systemctl start nginx
systemctl stop nginx
systemctl restart nginx
systemctl reload nginx                        # re-read config without a full restart, if the service supports it

reload only works if the service's unit defines an ExecReload — many services fall back to a full restart if it doesn't, so check systemctl status output for whether reload actually did what you expected.

Enabling and disabling on boot#

systemctl enable nginx                         # start automatically on boot
systemctl disable nginx
systemctl is-enabled nginx
systemctl enable --now nginx                    # enable AND start in one command

See the IMPORTANT callout at the top of this page — enable/disable (boot-time) and start/stop (running-now) are independent axes, and conflating them is the most common on-call mistake with this pair of commands.

Listing and filtering units#

systemctl list-units --type=service --state=running
systemctl list-units --state=failed             # everything currently in a failed state
systemctl is-active nginx
systemctl is-failed nginx

Masking a unit (stronger than disable)#

systemctl mask nginx      # prevents starting even manually or as a dependency of another unit
systemctl unmask nginx

Editing a unit with a drop-in#

systemctl edit creates an override on top of the vendor-supplied unit instead of modifying it directly, so a package update never silently clobbers your customization.

systemctl edit nginx                              # opens $EDITOR on a drop-in override.conf for the unit
systemctl edit --full nginx                        # edit a full copy of the unit instead of a drop-in
systemctl edit --drop-in=timeout.conf nginx         # use a named drop-in file instead of the default override.conf
systemctl edit --force nginx-custom.service          # create the unit if it doesn't exist yet
systemctl cat nginx                                   # show the effective unit file plus every drop-in applied to it

Saving in systemctl edit automatically reloads systemd config (equivalent to daemon-reload), but it does not restart the unit — a running service keeps running on its old config until you systemctl restart it.

Working with systemd timers#

.timer units are the systemd-native alternative to cron entries — they can fire on a fixed schedule (OnCalendar=) or relative to boot/last-run (OnBootSec=/OnUnitActiveSec=), and each run is captured in systemctl status/journalctl like any other unit.

systemctl list-timers                              # every active timer: next/last run time and its target unit
systemctl list-timers --all                          # include inactive/disabled timers too
systemctl status backup.timer                          # a timer's own status (separate from the service it triggers)
systemctl start backup.service                            # manually fire the timer's target service now, bypassing the schedule

A timer unit (backup.timer) and the unit it triggers (backup.service) are two separate units — systemctl status backup.timer only shows the schedule; check backup.service's own status/logs to see what happened during the actual run.

Reloading systemd after editing a unit file#

systemctl daemon-reload

Required any time you hand-edit a .service file — without it, systemd keeps using its in-memory copy of the old unit definition, and a restart will silently apply the old config.

Reading logs for a specific service#

journalctl -u nginx
journalctl -u nginx -f                          # follow/stream live
journalctl -u nginx --since "1 hour ago"
journalctl -u nginx --since "2026-08-19 09:00" --until "2026-08-19 10:00"
journalctl -u nginx -n 100                       # last 100 lines

Filtering and searching logs#

journalctl -p err                                # priority filter: emerg/alert/crit/err/warning/notice/info/debug
journalctl -g "connection refused"                # grep the message field
journalctl -b                                     # logs since the current boot only
journalctl -k                                      # kernel messages (dmesg equivalent, from the journal)
journalctl -o json-pretty -u nginx -n 5            # structured output for piping into another tool

Managing journal disk usage#

journalctl --disk-usage
journalctl --vacuum-time=7d          # delete journal entries older than 7 days
journalctl --vacuum-size=500M         # shrink the journal to under 500MB

An unbounded journal is a real, recurring cause of a host quietly running out of disk — --vacuum-time/--vacuum-size are the direct fix; the durable fix is setting SystemMaxUse= in /etc/systemd/journald.conf so it never grows unbounded in the first place.

Real-world scenario: a service stuck in a restart loop#

A service that keeps crashing shortly after start needs both why it crashed (the log) and how systemd is reacting to the crashes (the unit status) to diagnose fully:

systemctl status my-app                       # check Active: line for "activating (auto-restart)" — the tell-tale sign
journalctl -u my-app --since "10 min ago"      # what actually happened right before each crash
journalctl -u my-app -p err                     # filter straight to error-priority lines, skip the noise

Real-world scenario: correlating a kernel event with an application crash#

An application dies with no application-level log explaining why — often the kernel's own log (OOM killer, a hardware error) has the real answer, and journalctl merges both sources so they can be correlated by time:

journalctl -k --since "5 min ago"                 # kernel-only messages in the relevant window
journalctl -u my-app --since "5 min ago"            # the app's own log for the same window, side by side
journalctl -k -g "Out of memory"                     # search kernel messages specifically for an OOM event

An Out of memory: Killed process <pid> (my-app) kernel message correlated to the exact moment the application's own log goes silent is the standard way to confirm an OOM kill (not an application bug) caused a crash — checking dmesg/kernel logs is a required step, not optional, whenever an application "just dies" with no application-level error.

Common pitfalls#

• Assuming stop also disables boot-time startup — see the IMPORTANT callout at the top.
• Forgetting daemon-reload after hand-editing a unit file — restart silently applies the old config without it.
• Reading systemctl status's summary without checking the specific Active: state string — see the restart-loop scenario above; "is it running" and "is it stable" are different questions.
• Not checking kernel/OOM logs (journalctl -k) when an app "just dies" with nothing in its own log — see the second real-world scenario above.

When to reach for something else#

For distributed log aggregation across many hosts (not just one host's local journal), reach for a real log pipeline (see this site's Observability tutorials on the Elastic Stack and log management) — journalctl's scope is deliberately local-host-only, which is exactly right for a single-host incident but the wrong tool once "which of our 200 hosts logged this error" is the actual question.