Verified8 commandsAI-assisted

Debugging & Troubleshooting

Verified against kubectl v1.34.0 (client), flags verified via `kubectl <cmd> --help` run locally, 2026-08-29 · official docs

Events, resource usage, copying files, rollout status/rollback, ephemeral debug containers, and discovering what the API server actually supports — the toolkit for "why is this pod broken." 🔍

A mental model for triage order#

Diagram

Almost every "pod is broken" investigation starts the same way: get pods for the phase/reason, then branch based on what it says — the rest of this page's commands are how you dig into whichever branch applies.

Events#

kubectl get events                                        # events in the current namespace
kubectl get events --sort-by='{.lastTimestamp}'            # oldest-to-newest, so the latest is at the bottom
kubectl get events --field-selector involvedObject.name=my-pod
kubectl get events --field-selector type=Warning           # only warnings/errors
kubectl get events -A --field-selector type=Warning          # warnings across every namespace at once

kubectl get events is unsorted by default and truncates to the last hour by cluster policy in most setups — --sort-by and --field-selector are what make it actually useful instead of a wall of noise.

Resource usage#

kubectl top pod                             # CPU/memory for all pods in the current namespace
kubectl top pod --containers                # break down by container within each pod
kubectl top pod -l app=nginx
kubectl top node                            # CPU/memory for cluster nodes
kubectl top node --sort-by=cpu               # find the busiest node first

Note

top requires the metrics-server add-on running in the cluster — if it returns "error: Metrics API not available," that's a cluster configuration gap, not a typo in your command. It's also a point-in-time snapshot, not a history — for trend data over time, this is where the cluster's real metrics stack (Prometheus, per this site's Observability tutorials) takes over from kubectl top.

Copying files to/from a pod#

kubectl cp ./local-file.txt my-namespace/my-pod:/tmp/local-file.txt
kubectl cp my-namespace/my-pod:/var/log/app.log ./app.log
kubectl cp ./local-file.txt my-namespace/my-pod:/tmp/local-file.txt -c my-container   # target one container in a multi-container pod

kubectl cp requires tar to exist inside the target container's image — a distroless or scratch-based image will fail silently-ish with a tar-not-found error. kubectl exec ... -- tar | tar (piping through exec directly) is the fallback when the image has no tar binary.

Rollout status and rollback#

kubectl rollout status deployment/my-deployment            # watch a rollout until it completes or fails
kubectl rollout history deployment/my-deployment            # list revisions
kubectl rollout history deployment/my-deployment --revision=3   # what changed in a specific revision
kubectl rollout undo deployment/my-deployment                # roll back to the previous revision
kubectl rollout undo deployment/my-deployment --to-revision=3
kubectl rollout pause deployment/my-deployment                # freeze a rollout mid-flight — useful before making several related changes
kubectl rollout resume deployment/my-deployment

Tip

rollout pause/resume lets you batch several changes (image + resource limits + env vars) into one rollout instead of triggering a separate rolling update per change. Pause, make all the edits, resume — the deployment controller only starts actually rolling pods once resumed, so intermediate edits never each trigger their own partial rollout.

Ephemeral debug containers#

kubectl debug my-pod -it --image=busybox                        # attach a debug container to a running pod
kubectl debug my-pod -it --image=busybox --copy-to=my-pod-debug  # debug on a copy instead of the live pod
kubectl debug node/my-node -it --image=busybox                   # debug a node directly
kubectl debug my-pod -it --image=busybox --target=my-container    # share the target container's process namespace specifically

kubectl debug solves the distroless-image problem from a different angle than cp/exec: instead of needing shell/tar tools already inside the target container, it attaches a separate debug container (with whatever tools you choose) sharing the same pod/process namespace — the standard way to inspect a minimal production image without rebuilding it with debug tools baked in.

Important

--copy-to is the safer default for a production pod. Attaching a debug container directly to a live pod (no --copy-to) modifies that pod's spec while it's serving traffic; --copy-to clones the pod first and debugs the clone, leaving the original completely untouched — worth the extra flag on anything you're not 100% sure is safe to mutate live.

Discovering what resources exist#

kubectl api-resources                                    # every resource kind the API server supports, with short names
kubectl api-resources --namespaced=true                   # namespaced kinds only
kubectl api-resources --api-group=rbac.authorization.k8s.io   # scoped to one API group
kubectl api-resources -o wide                              # + which verbs (get/list/watch/...) each kind supports

kubectl api-versions                                       # every group/version the server has enabled, e.g. apps/v1

api-resources is the fastest way to find a resource's short name (kubectl get po instead of pods) or confirm a CRD actually registered — if kubectl get <thing> errors with "the server doesn't have a resource type," check it against api-resources before assuming a typo.

Real-world scenario: diagnosing a Pending pod#

A pod stuck in Pending almost always means the scheduler can't place it — the events explain why, but only if you look at the right resource's events:

kubectl get pod my-pod -o wide                    # confirm it's actually Pending, and shows no node assigned
kubectl describe pod my-pod                        # Events section: "0/5 nodes are available: 3 Insufficient cpu, 2 node(s) had taint..."
kubectl get nodes -o custom-columns=NAME:.metadata.name,CPU:.status.allocatable.cpu,MEM:.status.allocatable.memory
kubectl describe node my-node                       # confirm taints/allocatable resources on a specific candidate node

Warning

A Pending pod's own events almost always name the exact reason ("Insufficient cpu," "node(s) had taint {...} that the pod didn't tolerate," "didn't match Pod's node affinity/selector") — the mistake is looking anywhere else first. Jumping straight to checking node health or cluster capacity dashboards before reading kubectl describe pod's Events section is the single most common way this kind of investigation takes far longer than it needs to.

Real-world scenario: a CrashLoopBackOff that only reproduces under load#

An app that's fine at startup but crashes under real traffic needs the previous crashed instance's logs, not the currently-restarting one's (which hasn't crashed yet):

kubectl get pod my-pod -w                                  # watch the restart count climb in real time
kubectl logs my-pod --previous                               # the crashed instance's logs — where the actual error is
kubectl logs my-pod --previous --tail=200                     # if the previous instance logged a lot before crashing
kubectl describe pod my-pod                                   # check "Last State: Terminated, Reason: OOMKilled" specifically — a very common load-triggered crash cause

OOMKilled specifically means the container hit its memory limit (not the node's total memory) — the fix is either raising resources.limits.memory or finding and fixing the actual memory growth, not a scheduling or node-capacity issue at all.

Common pitfalls#

Debugging node/cluster capacity before reading describe pod's Events — see the WARNING above.
Checking kubectl logs right after a restart and seeing nothing — reach for --previous (see the Core Resources page too).
Attaching a debug container directly to a sensitive production pod without --copy-to — see the IMPORTANT callout above.
Assuming kubectl top shows historical trend data — it's a live snapshot only.

When to reach for something else#

For anything needing history, alerting, or correlation across many pods/nodes over time — not just "what's happening right now" — reach for the cluster's real observability stack (Prometheus/Grafana, or a managed equivalent) rather than trying to make kubectl top/logs do a job they're not built for. kubectl is the right tool for point-in-time inspection and direct intervention, not trend analysis.

← PreviousNetworking & Config Next →Cluster Administration & Advanced Usage