Network & Process Inspection
.mdVerified against ss (iproute2, Ubuntu 24.04), lsof 4.95.0, strace 6.8, tcpdump 4.99.4, iptables 1.8.10 (nf_tables), nftables 1.0.9, dig/nslookup 9.18.39, sysstat (iostat) 12.6.1, vmstat (procps) — all flags verified via `<cmd> --help`/`<cmd> -h`, `man strace`, and `man nft` run locally; netstat verified via man7.org/linux/man-pages/man8/netstat.8.html only (net-tools not installed in this environment), 2026-08-29 · official docs
What this page is 🎯#
Finding what's listening on a port, what a process has open, what syscalls it's making, and what's actually
on the wire — the toolkit that picks up once systemctl status/journalctl (the companion page) confirm a
service is running but something is still wrong at the network or OS level. These are deliberately many small,
single-purpose tools rather than one broad one — each answers a narrow question fast, and real troubleshooting
usually chains several of them together in sequence.
A triage order for "the service is up but unreachable"#
Sockets and listening ports — ss (modern) and netstat (legacy)#
ss -tulpn # all TCP+UDP listening sockets, with PID/program, numeric ports
ss -tan state established # established TCP connections only
ss -s # summary counts by protocol/state
netstat -tulpn # the older equivalent of the ss command above
netstat -r # kernel routing tabless is the current tool — it reads directly from the kernel and is significantly faster on a host with many connections; netstat (from net-tools) is legacy and not installed by default on many modern distros, including this one. Know both: ss for anything you run yourself, netstat for reading someone else's old runbook or a minimal/legacy host where ss isn't available either.
What's using a file, port, or directory — lsof#
lsof -i :8080 # what process (if any) is bound to port 8080
lsof -p 12345 # every file descriptor a specific PID has open
lsof -u myuser # every open file belonging to a user
lsof +D /var/log # every process with an open file under a directorylsof -i :PORT is usually the fastest way to answer "what's already using this port" when a service fails to bind on startup — faster than cross-referencing ss -tlpn output by eye.
Tracing syscalls — strace#
strace -p 12345 # attach to a running process and watch its syscalls live
strace -f -p 12345 # + follow any child processes it forks
strace -c -p 12345 # summary: syscall counts and time spent, not a live stream
strace -e trace=network -p 12345 # only network-related syscalls (connect, accept, sendto, ...)
strace -tt -o trace.log myprogram arg1 # run a fresh command under strace, with timestamps, to a filestrace adds real overhead to the traced process — fine for a one-off diagnostic attach, but not something to leave running against a production process under load without deciding that tradeoff deliberately first.
Capturing packets — tcpdump#
tcpdump -i eth0 # capture on a specific interface
tcpdump -i eth0 -n # -n: don't resolve hostnames (faster, avoids DNS noise in output)
tcpdump -i eth0 port 443
tcpdump -i eth0 host 10.0.1.5 and port 443
tcpdump -i eth0 -w capture.pcap # write raw packets to a file for later analysis (e.g. in Wireshark)
tcpdump -i eth0 -c 100 -X # capture exactly 100 packets, with hex+ASCII payload dumpCapturing on the wrong interface (eth0 vs a container's veth vs lo) is the most common reason "tcpdump shows nothing" during an actual incident — tcpdump -D lists every available interface if you're not sure which one carries the traffic you're chasing.
Blocking and inspecting traffic — iptables and nftables#
iptables is the legacy packet-filtering interface; nftables (nft) is its modern replacement. On this host iptables -V reports nf_tables as the backend, meaning both tools ultimately manage the same underlying kernel ruleset.
iptables -L -n -v # list all rules in the filter table, numeric, with packet/byte counters
iptables -L -n --line-numbers # same, with rule numbers (needed to target -D by position)
iptables -A INPUT -p tcp --dport 22 -j ACCEPT # append a rule to the INPUT chain
iptables -D INPUT 3 # delete rule #3 from INPUT
iptables -P INPUT DROP # change a chain's default policy
nft list ruleset # show the entire nftables ruleset, all tables/chains
nft add table inet mytable # create a new table (inet = both IPv4 and IPv6)
nft add chain inet mytable input '{ type filter hook input priority filter ; }'
nft add rule inet mytable input tcp dport 22 accept # add a rule to the chain
nft -a list ruleset # include rule handles, needed to target a specific rule for deletionDon't assume every host's iptables is nf_tables-backed like this one — some distros still ship the legacy iptables-legacy backend, where iptables and nft manage genuinely separate rulesets that can't see each other. Check iptables -V first.
DNS lookups — dig and nslookup#
dig example.com # full DNS answer, authority, and additional sections
dig example.com +short # just the answer, one line
dig example.com MX # query a specific record type
dig @8.8.8.8 example.com # query a specific nameserver directly, bypassing local resolver config
dig -x 93.184.216.34 # reverse lookup (PTR record)
dig example.com +trace # trace the full delegation path from the root nameservers down
nslookup example.com # quick forward lookup using the system resolver
nslookup example.com 8.8.8.8 # query a specific nameserverdig is the more capable tool for real troubleshooting — it exposes TTLs, which nameserver actually answered, and +trace for delegation problems. nslookup is faster to type for a quick sanity check but BIND's own docs point users toward dig/host instead.
System-wide performance — iostat and vmstat#
iostat -x 2 # extended per-device stats (%util, await, queue depth), every 2s
iostat -d -x sda 5 3 # extended stats for one device only, every 5s, 3 samples
vmstat 2 5 # 5 samples of memory/swap/IO/CPU summary, 2s apart
vmstat -a # active vs inactive memory instead of the default free/buff/cache split
vmstat -s # cumulative event counters since boot, not a live sampleiostat -x's %util column is usually the fastest way to tell "is this disk actually the bottleneck" — it approaches 100% when the device is saturated, regardless of raw throughput, which kB/s numbers alone don't tell you on their own.
Deeper strace usage#
strace -e trace=%file -p 12345 # only file-related syscalls (open, stat, unlink, ...)
strace -T -p 12345 # show time spent inside each syscall
strace -y -p 12345 # resolve file descriptors to their paths inline in the output
strace -c -e trace=%file myprogram # syscall-count summary, filtered to the file group, for a fresh command-e trace=%GROUP (e.g. %file, %network, %process, %signal) is the fast way to cut a noisy trace down to the syscall class you actually care about — check strace -e trace=? (or the strace(1) man page) for the full, current group list, since group membership has changed across strace releases.
Inspecting a running process via /proc#
cat /proc/<pid>/status # human-readable state, memory, thread count, uid/gid
cat /proc/<pid>/limits # the process's actual resolved ulimits
ls -l /proc/<pid>/fd # every open file descriptor, as symlinks to what they point to
cat /proc/<pid>/cmdline | tr '\0' ' ' # exact command line it was started with (NUL-separated, tr fixes it for display)
cat /proc/<pid>/environ | tr '\0' '\n' # its environment variables at start time
cat /proc/meminfo # system-wide memory stats (what free/vmstat parse)
cat /proc/loadavg # the 1/5/15-minute load averages/proc/<pid>/limits is the one to check when a process is hitting a "too many open files" or similar resource error — it shows the limits actually in effect for that specific process, which can differ from your own shell's ulimit -a if the process was started by a different parent (systemd, cron, a container runtime) with its own limits configured.
Real-world scenario: full triage of "service up, but unreachable"#
Following the diagram at the top of this page end to end, on a real incident:
ss -tlpn | grep :8080 # step 1: is anything actually bound to the port?
lsof -i :8080 # step 2: confirm it's the process you expect, not a stray old instance
iptables -L -n -v | grep 8080 # step 3: any firewall rule blocking it?
nft list ruleset | grep 8080 # (if this host uses nftables instead of/alongside iptables)
tcpdump -i eth0 -n port 8080 -c 20 # step 4: are packets even arriving at the interface at all?Warning
A surprisingly common root cause: an old, stale process still holding the port from a previous deploy
that didn't shut down cleanly. lsof -i :8080 showing a PID that doesn't match the current deployment's
expected process (check its start time with ps -o lstart= -p <pid>) is the tell — the "new" service never
actually started because bind failed silently or was masked by a supervisor retry loop, and traffic is
still hitting the old, possibly-broken instance.
Real-world scenario: diagnosing a slow endpoint with strace + tcpdump together#
An endpoint is slow, and it's unclear whether the delay is inside the application, in a downstream call, or on the network:
strace -T -f -p <pid> -e trace=network 2>&1 | grep -A1 "connect\|recvfrom" # -T shows time spent in each syscall
tcpdump -i eth0 -n host <downstream-ip> -w slow-request.pcap # capture the actual wire traffic for the same windowTip
strace -T (time spent per syscall) run alongside a tcpdump capture of the same window is how you tell
"slow in the app" from "slow on the network" without guessing. A recvfrom syscall that took 4 seconds
combined with a tcpdump capture showing the response packet actually arrived within milliseconds points at
application-side processing delay, not network latency — and vice versa.
Common pitfalls#
- Capturing on the wrong interface with tcpdump — see the note in the Capturing packets section;
-Dlists every available interface when it's not obvious which one carries the traffic. - Assuming every host's
iptablesis nf_tables-backed — see the iptables/nftables note; some hosts still run the legacy backend where the two tools manage genuinely separate rulesets. - Chasing an application bug when a stale process is actually holding the port — see the WARNING above;
always confirm the PID
lsof -ireports is actually the current deployment before debugging the app itself. - Leaving
straceattached to a production process under real load — it adds real overhead; treat it as a deliberate, time-boxed diagnostic attach, not something to leave running.
When to reach for something else#
For trend data over time (not point-in-time snapshots) — CPU/memory/disk/network history across a fleet, not just one host right now — reach for the cluster's real metrics stack (Prometheus/Grafana, per this site's Observability tutorials). These tools are the right choice for live, single-host, "what's happening right now" investigation; they don't retain history once the command exits.