# Linux Troubleshooting Cheat Sheet — Network & Process Inspection > **Tool:** ss, netstat, lsof, strace, tcpdump, iptables, nft, dig, nslookup, iostat, vmstat > **Category:** Linux & Troubleshooting > **Verified against:** ss (iproute2, Ubuntu 24.04), lsof 4.95.0, strace 6.8, tcpdump 4.99.4, iptables 1.8.10 (nf_tables), nftables 1.0.9, dig/nslookup 9.18.39, sysstat (iostat) 12.6.1, vmstat (procps) — all flags verified via ` --help`/` -h`, `man strace`, and `man nft` run locally; netstat verified via man7.org/linux/man-pages/man8/netstat.8.html only (net-tools not installed in this environment), 2026-08-29 > **Official docs:** https://man7.org/linux/man-pages/ ## What this page is 🎯 Finding what's listening on a port, what a process has open, what syscalls it's making, and what's actually on the wire — the toolkit that picks up once `systemctl status`/`journalctl` (the companion page) confirm a service is running but something is still wrong at the network or OS level. These are deliberately many small, single-purpose tools rather than one broad one — each answers a narrow question fast, and real troubleshooting usually chains several of them together in sequence. ## A triage order for "the service is up but unreachable" ```mermaid flowchart TD A["Service reported Active,
but clients can't connect"] --> B{"ss -tlpn: is it
actually bound to the port?"} B -->|"not listening"| C["App-level bind failure —
check its own logs"] B -->|"listening"| D{"lsof -i :PORT confirms
the right process owns it"} D --> E{"iptables/nft: any rule
blocking the port?"} E -->|"blocked"| F["Firewall rule is the cause"] E -->|"not blocked"| G["tcpdump on the interface —
are packets even arriving?"] classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c class C,F crit class G warn ``` ## Sockets and listening ports — ss (modern) and netstat (legacy) ```bash ss -tulpn # all TCP+UDP listening sockets, with PID/program, numeric ports ss -tan state established # established TCP connections only ss -s # summary counts by protocol/state netstat -tulpn # the older equivalent of the ss command above netstat -r # kernel routing table ``` `ss` is the current tool — it reads directly from the kernel and is significantly faster on a host with many connections; `netstat` (from `net-tools`) is legacy and not installed by default on many modern distros, including this one. Know both: `ss` for anything you run yourself, `netstat` for reading someone else's old runbook or a minimal/legacy host where `ss` isn't available either. ## What's using a file, port, or directory — lsof ```bash lsof -i :8080 # what process (if any) is bound to port 8080 lsof -p 12345 # every file descriptor a specific PID has open lsof -u myuser # every open file belonging to a user lsof +D /var/log # every process with an open file under a directory ``` `lsof -i :PORT` is usually the fastest way to answer "what's already using this port" when a service fails to bind on startup — faster than cross-referencing `ss -tlpn` output by eye. ## Tracing syscalls — strace ```bash strace -p 12345 # attach to a running process and watch its syscalls live strace -f -p 12345 # + follow any child processes it forks strace -c -p 12345 # summary: syscall counts and time spent, not a live stream strace -e trace=network -p 12345 # only network-related syscalls (connect, accept, sendto, ...) strace -tt -o trace.log myprogram arg1 # run a fresh command under strace, with timestamps, to a file ``` `strace` adds real overhead to the traced process — fine for a one-off diagnostic attach, but not something to leave running against a production process under load without deciding that tradeoff deliberately first. ## Capturing packets — tcpdump ```bash tcpdump -i eth0 # capture on a specific interface tcpdump -i eth0 -n # -n: don't resolve hostnames (faster, avoids DNS noise in output) tcpdump -i eth0 port 443 tcpdump -i eth0 host 10.0.1.5 and port 443 tcpdump -i eth0 -w capture.pcap # write raw packets to a file for later analysis (e.g. in Wireshark) tcpdump -i eth0 -c 100 -X # capture exactly 100 packets, with hex+ASCII payload dump ``` Capturing on the wrong interface (`eth0` vs a container's `veth` vs `lo`) is the most common reason "tcpdump shows nothing" during an actual incident — `tcpdump -D` lists every available interface if you're not sure which one carries the traffic you're chasing. ## Blocking and inspecting traffic — iptables and nftables iptables is the legacy packet-filtering interface; nftables (`nft`) is its modern replacement. On this host `iptables -V` reports `nf_tables` as the backend, meaning both tools ultimately manage the same underlying kernel ruleset. ```bash iptables -L -n -v # list all rules in the filter table, numeric, with packet/byte counters iptables -L -n --line-numbers # same, with rule numbers (needed to target -D by position) iptables -A INPUT -p tcp --dport 22 -j ACCEPT # append a rule to the INPUT chain iptables -D INPUT 3 # delete rule #3 from INPUT iptables -P INPUT DROP # change a chain's default policy nft list ruleset # show the entire nftables ruleset, all tables/chains nft add table inet mytable # create a new table (inet = both IPv4 and IPv6) nft add chain inet mytable input '{ type filter hook input priority filter ; }' nft add rule inet mytable input tcp dport 22 accept # add a rule to the chain nft -a list ruleset # include rule handles, needed to target a specific rule for deletion ``` Don't assume every host's `iptables` is nf_tables-backed like this one — some distros still ship the legacy `iptables-legacy` backend, where `iptables` and `nft` manage genuinely separate rulesets that can't see each other. Check `iptables -V` first. ## DNS lookups — dig and nslookup ```bash dig example.com # full DNS answer, authority, and additional sections dig example.com +short # just the answer, one line dig example.com MX # query a specific record type dig @8.8.8.8 example.com # query a specific nameserver directly, bypassing local resolver config dig -x 93.184.216.34 # reverse lookup (PTR record) dig example.com +trace # trace the full delegation path from the root nameservers down nslookup example.com # quick forward lookup using the system resolver nslookup example.com 8.8.8.8 # query a specific nameserver ``` `dig` is the more capable tool for real troubleshooting — it exposes TTLs, which nameserver actually answered, and `+trace` for delegation problems. `nslookup` is faster to type for a quick sanity check but BIND's own docs point users toward `dig`/`host` instead. ## System-wide performance — iostat and vmstat ```bash iostat -x 2 # extended per-device stats (%util, await, queue depth), every 2s iostat -d -x sda 5 3 # extended stats for one device only, every 5s, 3 samples vmstat 2 5 # 5 samples of memory/swap/IO/CPU summary, 2s apart vmstat -a # active vs inactive memory instead of the default free/buff/cache split vmstat -s # cumulative event counters since boot, not a live sample ``` `iostat -x`'s `%util` column is usually the fastest way to tell "is this disk actually the bottleneck" — it approaches 100% when the device is saturated, regardless of raw throughput, which kB/s numbers alone don't tell you on their own. ## Deeper strace usage ```bash strace -e trace=%file -p 12345 # only file-related syscalls (open, stat, unlink, ...) strace -T -p 12345 # show time spent inside each syscall strace -y -p 12345 # resolve file descriptors to their paths inline in the output strace -c -e trace=%file myprogram # syscall-count summary, filtered to the file group, for a fresh command ``` `-e trace=%GROUP` (e.g. `%file`, `%network`, `%process`, `%signal`) is the fast way to cut a noisy trace down to the syscall class you actually care about — check `strace -e trace=?` (or the strace(1) man page) for the full, current group list, since group membership has changed across strace releases. ## Inspecting a running process via /proc ```bash cat /proc//status # human-readable state, memory, thread count, uid/gid cat /proc//limits # the process's actual resolved ulimits ls -l /proc//fd # every open file descriptor, as symlinks to what they point to cat /proc//cmdline | tr '\0' ' ' # exact command line it was started with (NUL-separated, tr fixes it for display) cat /proc//environ | tr '\0' '\n' # its environment variables at start time cat /proc/meminfo # system-wide memory stats (what free/vmstat parse) cat /proc/loadavg # the 1/5/15-minute load averages ``` `/proc//limits` is the one to check when a process is hitting a "too many open files" or similar resource error — it shows the limits actually in effect for that specific process, which can differ from your own shell's `ulimit -a` if the process was started by a different parent (systemd, cron, a container runtime) with its own limits configured. ## Real-world scenario: full triage of "service up, but unreachable" Following the diagram at the top of this page end to end, on a real incident: ```bash ss -tlpn | grep :8080 # step 1: is anything actually bound to the port? lsof -i :8080 # step 2: confirm it's the process you expect, not a stray old instance iptables -L -n -v | grep 8080 # step 3: any firewall rule blocking it? nft list ruleset | grep 8080 # (if this host uses nftables instead of/alongside iptables) tcpdump -i eth0 -n port 8080 -c 20 # step 4: are packets even arriving at the interface at all? ``` > [!WARNING] > **A surprisingly common root cause: an old, stale process still holding the port from a previous deploy > that didn't shut down cleanly.** `lsof -i :8080` showing a PID that doesn't match the current deployment's > expected process (check its start time with `ps -o lstart= -p `) is the tell — the "new" service never > actually started because bind failed silently or was masked by a supervisor retry loop, and traffic is > still hitting the old, possibly-broken instance. ## Real-world scenario: diagnosing a slow endpoint with strace + tcpdump together An endpoint is slow, and it's unclear whether the delay is inside the application, in a downstream call, or on the network: ```bash strace -T -f -p -e trace=network 2>&1 | grep -A1 "connect\|recvfrom" # -T shows time spent in each syscall tcpdump -i eth0 -n host -w slow-request.pcap # capture the actual wire traffic for the same window ``` > [!TIP] > **`strace -T` (time spent per syscall) run alongside a `tcpdump` capture of the same window is how you tell > "slow in the app" from "slow on the network" without guessing.** A `recvfrom` syscall that took 4 seconds > combined with a `tcpdump` capture showing the response packet actually arrived within milliseconds points at > application-side processing delay, not network latency — and vice versa. ## Common pitfalls - **Capturing on the wrong interface with tcpdump** — see the note in the Capturing packets section; `-D` lists every available interface when it's not obvious which one carries the traffic. - **Assuming every host's `iptables` is nf_tables-backed** — see the iptables/nftables note; some hosts still run the legacy backend where the two tools manage genuinely separate rulesets. - **Chasing an application bug when a stale process is actually holding the port** — see the WARNING above; always confirm the PID `lsof -i` reports is actually the current deployment before debugging the app itself. - **Leaving `strace` attached to a production process under real load** — it adds real overhead; treat it as a deliberate, time-boxed diagnostic attach, not something to leave running. ## When to reach for something else For trend data over time (not point-in-time snapshots) — CPU/memory/disk/network history across a fleet, not just one host right now — reach for the cluster's real metrics stack (Prometheus/Grafana, per this site's Observability tutorials). These tools are the right choice for live, single-host, "what's happening right now" investigation; they don't retain history once the command exits.