# Linux Troubleshooting Cheat Sheet — systemd & journalctl

> **Tool:** systemd (systemctl, journalctl)
> **Category:** Linux & Troubleshooting
> **Verified against:** systemd 255 (255.4-1ubuntu8.17), flags verified via `systemctl --help`, `journalctl --help`, and `man systemctl` run locally, 2026-08-29
> **Official docs:** https://www.man7.org/linux/man-pages/man1/systemctl.1.html, https://www.man7.org/linux/man-pages/man1/journalctl.1.html

## What it is and where it fits 🎯

systemd is the init system and service manager on essentially every modern Linux distribution — PID 1, the
first process the kernel starts, responsible for bringing up every other service in dependency order and
supervising them afterward. `systemctl` is the control surface (start/stop/enable/status a unit);
`journalctl` reads the structured, binary journal systemd's logging component (`journald`) writes to,
replacing the older plain-text `/var/log/*.log` + `syslog` model on most distros. These are the first two
tools for "why did this service stop / why won't it start" — everything else in Linux troubleshooting (network
inspection, process tracing) usually comes after confirming *whether the service is even running*.

## A unit's lifecycle, and where each command touches it

```mermaid
stateDiagram-v2
    [*] --> Inactive
    Inactive --> Activating: systemctl start
    Activating --> Active: unit reports ready
    Active --> Deactivating: systemctl stop
    Deactivating --> Inactive
    Active --> Failed: process crashes /<br/>exits non-zero
    Failed --> Activating: systemctl restart

    note right of Inactive
        enable/disable control
        BOOT-TIME startup,
        independent of this
        running-state machine
    end note
```

> [!IMPORTANT]
> **`enable`/`disable` and `start`/`stop` are two independent axes, not one.** `enable` controls whether a
> unit starts automatically on the *next* boot; `start` controls whether it's running *right now*. A service
> can be enabled but currently stopped, or running but not enabled (won't survive a reboot). The classic
> on-call mistake is `systemctl stop` during an incident without also `disable` — then being surprised the
> service is back after the next reboot or deploy re-triggers a boot-time start.

## Service status and control

```bash
systemctl status nginx
systemctl start nginx
systemctl stop nginx
systemctl restart nginx
systemctl reload nginx                        # re-read config without a full restart, if the service supports it
```

`reload` only works if the service's unit defines an `ExecReload` — many services fall back to a full restart if it doesn't, so check `systemctl status` output for whether reload actually did what you expected.

## Enabling and disabling on boot

```bash
systemctl enable nginx                         # start automatically on boot
systemctl disable nginx
systemctl is-enabled nginx
systemctl enable --now nginx                    # enable AND start in one command
```

See the IMPORTANT callout at the top of this page — `enable`/`disable` (boot-time) and `start`/`stop`
(running-now) are independent axes, and conflating them is the most common on-call mistake with this pair of
commands.

## Listing and filtering units

```bash
systemctl list-units --type=service --state=running
systemctl list-units --state=failed             # everything currently in a failed state
systemctl is-active nginx
systemctl is-failed nginx
```

## Masking a unit (stronger than disable)

```bash
systemctl mask nginx      # prevents starting even manually or as a dependency of another unit
systemctl unmask nginx
```

## Editing a unit with a drop-in

`systemctl edit` creates an override on top of the vendor-supplied unit instead of modifying it directly, so a package update never silently clobbers your customization.

```bash
systemctl edit nginx                              # opens $EDITOR on a drop-in override.conf for the unit
systemctl edit --full nginx                        # edit a full copy of the unit instead of a drop-in
systemctl edit --drop-in=timeout.conf nginx         # use a named drop-in file instead of the default override.conf
systemctl edit --force nginx-custom.service          # create the unit if it doesn't exist yet
systemctl cat nginx                                   # show the effective unit file plus every drop-in applied to it
```

Saving in `systemctl edit` automatically reloads systemd config (equivalent to `daemon-reload`), but it does **not** restart the unit — a running service keeps running on its old config until you `systemctl restart` it.

## Working with systemd timers

`.timer` units are the systemd-native alternative to cron entries — they can fire on a fixed schedule (`OnCalendar=`) or relative to boot/last-run (`OnBootSec=`/`OnUnitActiveSec=`), and each run is captured in `systemctl status`/`journalctl` like any other unit.

```bash
systemctl list-timers                              # every active timer: next/last run time and its target unit
systemctl list-timers --all                          # include inactive/disabled timers too
systemctl status backup.timer                          # a timer's own status (separate from the service it triggers)
systemctl start backup.service                            # manually fire the timer's target service now, bypassing the schedule
```

A timer unit (`backup.timer`) and the unit it triggers (`backup.service`) are two separate units — `systemctl status backup.timer` only shows the schedule; check `backup.service`'s own status/logs to see what happened during the actual run.

## Reloading systemd after editing a unit file

```bash
systemctl daemon-reload
```

Required any time you hand-edit a `.service` file — without it, systemd keeps using its in-memory copy of the old unit definition, and a `restart` will silently apply the *old* config.

## Reading logs for a specific service

```bash
journalctl -u nginx
journalctl -u nginx -f                          # follow/stream live
journalctl -u nginx --since "1 hour ago"
journalctl -u nginx --since "2026-08-19 09:00" --until "2026-08-19 10:00"
journalctl -u nginx -n 100                       # last 100 lines
```

## Filtering and searching logs

```bash
journalctl -p err                                # priority filter: emerg/alert/crit/err/warning/notice/info/debug
journalctl -g "connection refused"                # grep the message field
journalctl -b                                     # logs since the current boot only
journalctl -k                                      # kernel messages (dmesg equivalent, from the journal)
journalctl -o json-pretty -u nginx -n 5            # structured output for piping into another tool
```

## Managing journal disk usage

```bash
journalctl --disk-usage
journalctl --vacuum-time=7d          # delete journal entries older than 7 days
journalctl --vacuum-size=500M         # shrink the journal to under 500MB
```

An unbounded journal is a real, recurring cause of a host quietly running out of disk — `--vacuum-time`/`--vacuum-size` are the direct fix; the durable fix is setting `SystemMaxUse=` in `/etc/systemd/journald.conf` so it never grows unbounded in the first place.

## Real-world scenario: a service stuck in a restart loop

A service that keeps crashing shortly after start needs both *why it crashed* (the log) and *how systemd is
reacting to the crashes* (the unit status) to diagnose fully:

```bash
systemctl status my-app                       # check Active: line for "activating (auto-restart)" — the tell-tale sign
journalctl -u my-app --since "10 min ago"      # what actually happened right before each crash
journalctl -u my-app -p err                     # filter straight to error-priority lines, skip the noise
```

> [!TIP]
> **`systemctl status`'s `Active:` line distinguishes a genuinely stopped service from one systemd is
> actively fighting to keep alive.** `activating (auto-restart)` means the unit has a `Restart=` policy and is
> repeatedly crash-looping — very different from a cleanly `inactive (dead)` service someone stopped on
> purpose, and the two easily get conflated at a glance if you only check whether the service "is running"
> rather than reading the actual state string.

## Real-world scenario: correlating a kernel event with an application crash

An application dies with no application-level log explaining why — often the kernel's own log (OOM killer,
a hardware error) has the real answer, and `journalctl` merges both sources so they can be correlated by time:

```bash
journalctl -k --since "5 min ago"                 # kernel-only messages in the relevant window
journalctl -u my-app --since "5 min ago"            # the app's own log for the same window, side by side
journalctl -k -g "Out of memory"                     # search kernel messages specifically for an OOM event
```

An `Out of memory: Killed process <pid> (my-app)` kernel message correlated to the exact moment the
application's own log goes silent is the standard way to confirm an OOM kill (not an application bug) caused
a crash — checking `dmesg`/kernel logs is a required step, not optional, whenever an application "just dies"
with no application-level error.

## Common pitfalls

- **Assuming `stop` also disables boot-time startup** — see the IMPORTANT callout at the top.
- **Forgetting `daemon-reload` after hand-editing a unit file** — `restart` silently applies the *old* config
  without it.
- **Reading `systemctl status`'s summary without checking the specific `Active:` state string** — see the
  restart-loop scenario above; "is it running" and "is it stable" are different questions.
- **Not checking kernel/OOM logs (`journalctl -k`) when an app "just dies"** with nothing in its own log —
  see the second real-world scenario above.

## When to reach for something else

For distributed log aggregation across many hosts (not just one host's local journal), reach for a real log
pipeline (see this site's Observability tutorials on the Elastic Stack and log management) — `journalctl`'s
scope is deliberately local-host-only, which is exactly right for a single-host incident but the wrong tool
once "which of our 200 hosts logged this error" is the actual question.