# Operating Systems & Compute Fundamentals — Part 2: The Scheduler & Process Execution > **Series:** Operating Systems & Compute Fundamentals (2 of 7) > **Part 1:** `01-computer-architecture-and-the-cpu.md` — Computer Architecture & the CPU > **Part 2:** This file — The Scheduler & Process Execution > **Part 3:** `03-memory-management-deep-dive.md` — Memory Management Deep Dive > **Part 4:** `04-storage-and-filesystems.md` — Storage & Filesystems > **Part 5:** `05-virtualization-fundamentals.md` — Virtualization Fundamentals > **Part 6:** `06-boot-process-and-kernel-architecture.md` — The Boot Process & Kernel Architecture > **Part 7:** `07-resource-isolation-and-capacity-planning-in-practice.md` — Resource Isolation & Capacity Planning in Practice > **Questions:** `questions.md` > Assumes you're comfortable with Part 1's coverage of cores, context switches, and cache locality — this chapter is about how the kernel decides which thread gets those cores, for how long. ## Table of Contents 1. [What the Scheduler Actually Decides](#what-the-scheduler-actually-decides) 2. [Process States and the Run Queue](#process-states-and-the-run-queue) 3. [The Completely Fair Scheduler — Fairness via Virtual Runtime](#the-completely-fair-scheduler--fairness-via-virtual-runtime) 4. [EEVDF — The Scheduler Since Linux 6.6](#eevdf--the-scheduler-since-linux-66) 5. [Two Different Schedulers — kube-scheduler vs. the Linux Scheduler](#two-different-schedulers--kube-scheduler-vs-the-linux-scheduler) 6. [Nice Values and Priorities](#nice-values-and-priorities) 7. [Scheduling Classes — Beyond SCHED_NORMAL](#scheduling-classes--beyond-sched_normal) 8. [cgroups and CPU Quotas — How Kubernetes CPU Limits Actually Work](#cgroups-and-cpu-quotas--how-kubernetes-cpu-limits-actually-work) 9. [CPU Throttling — Why "Under the Limit" Still Stutters](#cpu-throttling--why-under-the-limit-still-stutters) 10. [CPU Requests vs. Limits — Shares vs. Hard Ceilings](#cpu-requests-vs-limits--shares-vs-hard-ceilings) 11. [Inside cpu.weight — How Kubernetes Requests Become a Real Scheduler Value](#inside-cpuweight--how-kubernetes-requests-become-a-real-scheduler-value) 12. [cgroup v1 vs. v2 — Why the File Paths and Semantics Differ](#cgroup-v1-vs-v2--why-the-file-paths-and-semantics-differ) 13. [Multi-Threaded Runtimes and the Scheduler — Why Language Choice Interacts With Throttling](#multi-threaded-runtimes-and-the-scheduler--why-language-choice-interacts-with-throttling) 14. [Wait Queues and Wakeups — How a Sleeping Thread Becomes Runnable Again](#wait-queues-and-wakeups--how-a-sleeping-thread-becomes-runnable-again) 15. [Priority Inversion — When a Low-Priority Thread Blocks a High-Priority One](#priority-inversion--when-a-low-priority-thread-blocks-a-high-priority-one) 16. [Load Average — What That Number Actually Means](#load-average--what-that-number-actually-means) 17. [Load Balancing Across Cores](#load-balancing-across-cores) 18. [Preemption and Scheduling Latency](#preemption-and-scheduling-latency) 19. [The Scheduler Tick — How Often the Kernel Even Gets a Chance to Reschedule](#the-scheduler-tick--how-often-the-kernel-even-gets-a-chance-to-reschedule) 20. [Voluntary vs. Involuntary Context Switches — What Each One Tells You](#voluntary-vs-involuntary-context-switches--what-each-one-tells-you) 21. [How CPU Affinity Constrains the Scheduler's Choices](#how-cpu-affinity-constrains-the-schedulers-choices) 22. [Reading Scheduler Behavior on a Real Machine](#reading-scheduler-behavior-on-a-real-machine) 23. [Garbage Collection as a Scheduling Event](#garbage-collection-as-a-scheduling-event) 24. [How These Concepts Show Up on the Cloud Bill](#how-these-concepts-show-up-on-the-cloud-bill) 25. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps) 26. [Worked Practice Problems](#worked-practice-problems) 27. [Summary and What's Next](#summary-and-whats-next) --- ## What the Scheduler Actually Decides Part 1 established that a CPU core executes exactly one instruction stream at a time, and that context switching is the mechanism letting many threads share it. This chapter covers the piece of the kernel actually responsible for deciding *which* thread gets a core *next*, and for *how long* — the **process scheduler**. On a typical `checkout-service` host running dozens of application threads alongside kernel worker threads, monitoring agents, and system daemons, the scheduler makes this decision continuously, many thousands of times per second, on every core, and gets almost no attention until it's the reason a service is slow. The running example remains this site's fictional e-commerce platform — `checkout-service`, `catalog-service`, `inventory-service` — the same throughline used throughout this series and across this site's Terraform, Kubernetes, Observability, and Incident Management series. > [!NOTE] > This chapter focuses on Linux's scheduler for `SCHED_NORMAL` (ordinary, non-real-time) tasks, since that's what the overwhelming majority of production application workloads run under. Section 7 briefly covers the other scheduling classes that exist alongside it. **Where this chapter goes from here:** it moves from the basic mechanics (process states, the CFS/EEVDF fairness algorithms) through the operator-facing controls built on top of them (nice values, scheduling classes, cgroup quotas), into the Kubernetes-specific translation of those controls (requests vs. limits, throttling), and closes with the deeper mechanisms — priority inversion, load balancing across cores, the scheduler tick — that explain the less obvious scheduling behavior a platform engineer eventually runs into in production. --- ## Process States and the Run Queue Every thread the kernel manages is, at any given moment, in one of a small number of well-defined states. The scheduler's entire job is choosing which **runnable** thread — one ready and waiting for a core — actually gets one next. ```mermaid stateDiagram-v2 [*] --> Runnable: Thread created or woken up Runnable --> Running: Scheduler selects it for a core Running --> Runnable: Time slice expires or preempted Running --> Sleeping: Blocks on I/O, a lock, or a syscall Sleeping --> Runnable: The event it was waiting for occurs Running --> UninterruptibleSleep: Blocks on I/O that cannot be interrupted (disk, NFS) UninterruptibleSleep --> Runnable: The I/O operation completes Running --> Zombie: Process exits, parent hasn't reaped it yet Zombie --> [*]: Parent calls wait(), resources freed classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c class Running ok class Runnable ok class UninterruptibleSleep crit class Zombie warn ``` **What to notice:** only **runnable** threads (state `R` in `top`/`ps`) compete for the scheduler's attention at all — a thread in interruptible sleep (`S`, waiting on a normal blocking call) or uninterruptible sleep (`D`, waiting on disk/NFS I/O that can't be safely interrupted) is entirely off the scheduler's radar until whatever it's waiting for actually happens. This single fact is the foundation for everything else in this chapter: the scheduler only ever has to arbitrate among threads that are genuinely ready to run right now, and the **run queue** — the per-core (in modern Linux, per-core, not global) data structure holding exactly those threads — is what it picks from. | State | `ps`/`top` code | What it means | Competes for a core? | |---|---|---|---| | Running | `R` | Actively executing, or in the run queue ready to be | Yes | | Interruptible sleep | `S` | Blocked on a normal event (a socket read, a timer, a signal) | No, until woken | | Uninterruptible sleep | `D` | Blocked on I/O the kernel won't interrupt (disk, NFS) — famously un-killable, even with `SIGKILL`, until the I/O resolves | No, until the I/O completes | | Stopped | `T` | Suspended by a signal (`SIGSTOP`) or a debugger | No, until resumed | | Zombie | `Z` | Exited, but its exit status hasn't been collected by its parent yet | No — it holds no CPU or memory resources, just a process-table entry | **From the Trenches:** an on-call engineer tried repeatedly to `kill -9` a hung `inventory-service` worker process during an incident, with no effect at all — the process simply stayed alive. The immediate cause, visible in `ps aux`'s state column, was that the process was in uninterruptible sleep (`D`), blocked on a network filesystem mount that had silently become unresponsive; `SIGKILL` cannot terminate a process stuck in this state, because the kernel has already committed to completing the I/O operation before it can safely tear the process down. The underlying, two-levels-deep condition was that the actual root cause was never the application process at all — it was a degraded NFS server the process's disk I/O depended on, and no amount of process-level intervention could fix a problem that lived one layer further down, in the storage path this series covers in Part 4. The fix wasn't killing the process; it was restoring the NFS mount's health, after which the process's I/O completed and it exited normally on its own. > [!TIP] > **Best practice:** before escalating a "process won't die" incident, check its state with `ps -o pid,stat,comm -p ` first. A `D` in the `STAT` column immediately redirects the investigation toward the underlying I/O path instead of wasting time on repeated kill attempts that were never going to succeed — a five-second check that avoids a genuinely common on-call time sink. --- ## The Completely Fair Scheduler — Fairness via Virtual Runtime For most of the past two decades, Linux's default scheduler for ordinary tasks was the **Completely Fair Scheduler (CFS)**, introduced in kernel 2.6.23. Its core idea is genuinely elegant: track how much CPU time each runnable thread has actually received, weighted by its priority, and always hand the core to whichever thread has received the *least* so far — making every thread's CPU time converge toward its fair share over time, rather than using fixed time slices handed out in a rigid round-robin. ```mermaid graph TD RQ["Run queue: threads ordered
by virtual runtime (vruntime)
in a red-black tree"] --> Pick["Scheduler always picks
the LEFTMOST node —
lowest vruntime"] Pick --> Run["That thread runs
until preempted or blocked"] Run --> Update["Its vruntime increases
by time actually run,
weighted by priority"] Update --> Reinsert["Re-inserted into the tree
at its new position"] Reinsert --> RQ classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c class Pick ok class RQ info ``` **What to notice:** CFS never needs a fixed "time slice" the way an older round-robin scheduler would — the red-black tree data structure keeps every runnable thread ordered by **vruntime** (virtual runtime, a priority-weighted measure of CPU time already consumed), and the scheduler's selection rule is always simply "run whichever thread is furthest to the left" — the one with the least accumulated fair-share time. A thread that runs for a while has its vruntime climb, naturally sorting it further right in the tree and making it progressively less likely to be picked again until the other threads catch up. **From the Trenches:** a team debugging why a low-priority background reindexing job for `catalog-service` seemed to be "stealing" more CPU time than expected from the foreground request-handling threads discovered, via `chrt` and process priority inspection, that both had been left at the identical default nice value. The immediate cause was a deployment-script oversight — the background job's `nice` level had been intended to be set to a lower priority at process launch, but the flag had been silently dropped during a recent refactor of the startup script. The underlying, two-levels-deep condition was that CFS was doing exactly what it's designed to do — treating two equal-priority threads with genuine fairness — but "fair" here meant equal, and the team's actual intent (foreground work should win) had never been expressed to the scheduler at all, so there was nothing for CFS to enforce. The fix was restoring the missing `nice` flag, which is covered in full in Section 6. --- ## EEVDF — The Scheduler Since Linux 6.6 Linux 6.6 (released October 2023) replaced CFS as the default scheduler for ordinary tasks with **EEVDF** (Earliest Eligible Virtual Deadline First) — a change worth knowing by name specifically because CFS had been the unquestioned default for so long that "CFS" and "the Linux scheduler" became functionally synonymous in a lot of operational vocabulary, and that's no longer accurate on a current kernel. EEVDF keeps much of CFS's underlying machinery — tasks are still tracked in a per-core structure ordered by a virtual-runtime-like measure — but changes the actual selection rule: rather than always picking the single lowest-vruntime thread, EEVDF assigns each thread an **eligible time** and a **virtual deadline**, and picks among threads that have become eligible to run, favoring the one with the earliest deadline. In practice, this produces measurably better latency and more predictable scheduling behavior for latency-sensitive, bursty workloads than CFS's simpler heuristics could reliably guarantee — CFS had, over its lifetime, accumulated a large number of tunable heuristics and special-cases to compensate for scenarios its original fairness model handled imperfectly, and EEVDF's design goal was addressing several of those root causes more directly rather than adding another heuristic on top. | | CFS (pre-6.6 default) | EEVDF (6.6+ default) | |---|---|---| | Core data structure | Red-black tree ordered by vruntime | Similar structure, ordered by virtual deadline | | Selection rule | Always the single lowest-vruntime thread | Earliest virtual deadline among eligible threads | | Latency-sensitive workload behavior | Good, but relied on multiple accumulated heuristics | Designed to handle this case more directly, with fewer special-case heuristics | | Practical visibility to an operator | `nice` values, cgroup CPU shares/quotas behave the same way from the outside | Same external interface — `nice`, cgroup shares/quotas all still apply identically | | First stable, non-experimental kernel | 6.6 (October 2023) | Absorbed further stabilization fixes through 6.8 (March 2024) before being considered fully settled | > [!NOTE] > The genuinely important operational takeaway: **nothing about how you configure CPU priorities or cgroup limits changes** between CFS and EEVDF — `nice` values, cgroup `cpu.weight`/`cpu.max`, and Kubernetes resource requests/limits all mean exactly the same thing to an operator under either scheduler. The difference is entirely in the kernel's own internal selection algorithm, which is exactly why this section exists as a "know the name, understand the reasoning" chapter section rather than a hands-on tuning one — there's nothing new to configure, only new vocabulary worth recognizing if it comes up in a kernel version changelog or an interview. **Interview-ready line:** "CFS scheduled the thread with the lowest accumulated, priority-weighted runtime next, using a red-black tree ordered by vruntime. EEVDF, the default since Linux 6.6, keeps similar underlying machinery but selects by earliest virtual deadline among eligible threads instead, aiming for more predictable latency with fewer accumulated heuristics. Neither change affects how an operator configures priority — nice values and cgroup CPU shares/quotas work identically under both." --- ## Two Different Schedulers — kube-scheduler vs. the Linux Scheduler A platform engineer working with Kubernetes deals with two entirely separate mechanisms that both happen to be called "the scheduler," and confusing them is a genuinely common source of miscommunication and misdiagnosis — worth untangling explicitly before going further into this chapter's Linux-kernel-specific mechanisms. ```mermaid flowchart TD Pod["New checkout-service Pod
submitted to the API server"] --> KubeSched["kube-scheduler:
WHICH NODE should
this pod run on?"] KubeSched -->|"considers resource requests,
affinity/anti-affinity, taints"| NodeChoice["Node chosen —
pod is bound to it"] NodeChoice --> Kubelet["kubelet on that node
starts the container"] Kubelet --> LinuxSched["Linux kernel scheduler:
WHICH CORE, WHEN, for how long
does each of this container's
THREADS actually run?"] classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c class KubeSched info class LinuxSched warn ``` **What to notice:** `kube-scheduler` operates at the **pod-to-node** granularity, running once per pod at placement time — it answers "which of the cluster's nodes has room for this pod's requested resources, and satisfies its affinity/taint/toleration rules?" and then never touches that pod again unless it's rescheduled. The **Linux kernel scheduler** covered throughout the rest of this chapter operates at the **thread-to-core** granularity, continuously, many thousands of times per second, for the entire lifetime of every process on that node — it's the mechanism that actually determines a running pod's real-time CPU experience, entirely invisible to `kube-scheduler` once placement is done. | | `kube-scheduler` | Linux kernel scheduler (CFS/EEVDF) | |---|---|---| | Decides | Which node a pod runs on | Which core a thread runs on, and for how long | | Runs | Once, at pod placement (and on explicit reschedule) | Continuously, thousands of times per second, on every core | | Inputs | Resource requests, node capacity, affinity rules, taints/tolerations | Nice values, cgroup CPU shares/quotas, thread run/block state | | A "scheduling problem" here means | A pod stuck `Pending`, or landing on a poorly-suited node | A running pod's threads experiencing throttling, unfair CPU share, or high scheduling latency | | The right tool to check first | `kubectl describe pod`, `kubectl get events`, `kube-scheduler` logs | `mpstat`, `cpu.stat`, `pidstat`, `vmstat` — this chapter's own diagnostic toolkit | | Who typically owns diagnosing it | Cluster/platform team, node capacity and scheduling constraints | Either team, depending on whether the cause is host-level noise or the workload's own resource configuration | **From the Trenches:** a report that "the scheduler is broken" for `checkout-service` turned into a half-day miscommunication between a platform team and an application team before anyone realized they were describing two completely different problems using the same word. The platform team, hearing "scheduler," investigated `kube-scheduler` logs and pod placement events, and found nothing wrong — the pod had been placed promptly on a healthy node. The application team's actual complaint, once clarified, was about *in-pod* latency stutters entirely consistent with the CFS-throttling pattern from Section 9, a Linux-kernel-scheduler problem `kube-scheduler` has no visibility into at all once placement is complete. The underlying, two-levels-deep condition was that "the scheduler" is genuinely ambiguous shorthand in a Kubernetes context specifically because two real, separate systems share the name, and neither team had thought to explicitly disambiguate which one they meant before spending hours investigating the wrong one. The fix, beyond resolving the actual throttling issue, was adopting "pod scheduler" and "kernel/CPU scheduler" as the team's standard disambiguated vocabulary in incident channels going forward. > [!TIP] > **Best practice:** when a Kubernetes-context "scheduling" issue is reported, clarify immediately whether it's a pod-placement problem (`kubectl describe pod` showing `Pending`/scheduling events, `kube-scheduler` logs) or an in-pod CPU-scheduling problem (throttling metrics, context-switch rates, `mpstat` — this chapter's actual subject) before investigating either. The two require entirely different tools and entirely different people's mental models. --- ## Nice Values and Priorities Every process has a **nice value**, an integer from -20 (highest priority) to +19 (lowest priority), that directly controls its weight in the scheduler's fairness calculation — a lower nice value means a thread's vruntime climbs more slowly relative to its peers, so it gets picked more often and for effectively larger shares of CPU time. ```mermaid graph LR Neg20["nice -20
(highest priority)"] --> Zero["nice 0
(default)"] Zero --> Pos19["nice +19
(lowest priority)"] classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c classDef muted fill:#eaeef1,stroke:#c3ccd4,color:#10161c class Neg20 crit class Zero info class Pos19 muted ``` | Command | Effect | |---|---| | `nice -n 10 ` | Launch a new process at a lower priority (nice value 10) | | `renice -n -5 -p ` | Change an already-running process's nice value (requires elevated privileges for negative values) | | `chrt -p ` | Show or set the scheduling *policy* (Section 7), distinct from nice value | The practical effect is a real, tunable lever: a batch job or background maintenance task can be deliberately deprioritized (a higher nice value) so it yields to foreground request-handling work whenever both are runnable, without needing to be stopped or scheduled at a different time entirely. It's a soft preference, not a hard guarantee — a niced-down process still gets *some* CPU time (CFS/EEVDF fairness never fully starves a runnable thread), just proportionally less of it under contention. > [!TIP] > **Best practice:** any long-running background or batch process sharing a host with latency-sensitive foreground services should be launched with an explicitly elevated nice value, not left at the default. This is a cheap, well-understood lever that avoids exactly the class of "why is my background job affecting foreground latency" incident from this chapter's earlier From the Trenches example — and it's a far lighter-weight fix than the cgroup-level isolation covered in the next two sections, appropriate when soft deprioritization, not hard isolation, is all that's actually needed. --- ## Scheduling Classes — Beyond SCHED_NORMAL CFS/EEVDF governs `SCHED_NORMAL`, the default policy essentially every application thread runs under — but Linux supports several distinct scheduling **classes**, each with entirely different guarantees, and a platform engineer should at least recognize them by name. | Scheduling class | Guarantee | Typical use | |---|---|---| | `SCHED_NORMAL` (also called `SCHED_OTHER`) | Fair-share, best-effort — the CFS/EEVDF policy this whole chapter has covered | The overwhelming majority of application and system processes | | `SCHED_BATCH` | Like `SCHED_NORMAL`, but the kernel assumes the thread isn't latency-sensitive and optimizes differently (less aggressive wake-up preemption) | CPU-bound batch/background jobs that shouldn't disturb interactive workloads | | `SCHED_IDLE` | Runs only when literally nothing else on the system is runnable | The lowest-priority background work imaginable — cleanup tasks that should never compete with anything | | `SCHED_FIFO` | Real-time, fixed priority, runs until it voluntarily yields or a higher-priority real-time thread preempts it — no fairness guarantee at all against lower-priority threads | Hard real-time requirements — rare in typical web-service infrastructure, common in specialized embedded/control systems | | `SCHED_RR` | Real-time, like `SCHED_FIFO` but with a fixed time-slice round-robin among equal-priority real-time threads | Similar real-time use cases needing round-robin fairness among peers at the same priority | | `SCHED_DEADLINE` | Real-time, but specified as an explicit runtime/period/deadline triple rather than a fixed priority — the kernel admits it only if it can guarantee the deadline | The strictest real-time guarantee available, for workloads that can express their exact timing requirements numerically | > [!WARNING] > A `SCHED_FIFO`/`SCHED_RR` real-time thread that misbehaves (an infinite loop, a bug that never yields) can genuinely starve every `SCHED_NORMAL` thread on that core indefinitely — real-time scheduling classes exist specifically to override the fairness guarantees this chapter otherwise relies on, which is exactly why they're reserved for genuinely time-critical, carefully audited code, not reached for as a casual "make this thread higher priority" tool. `nice` values (Section 6) are the correct, safe tool for that far more common case. --- ## cgroups and CPU Quotas — How Kubernetes CPU Limits Actually Work Nice values operate on individual processes, one at a time — **cgroups** (control groups) let the kernel apply CPU scheduling constraints to an entire *group* of processes at once, and this is the exact mechanism underneath every Kubernetes CPU request and limit, every Docker `--cpus` flag, and every systemd service's `CPUQuota=` setting. ```mermaid flowchart TD Pod["checkout-service Pod spec:
resources.limits.cpu: 2"] --> Kubelet["kubelet translates this
into a cgroup configuration"] Kubelet --> Cgroup["cgroup cpu.max file:
200000 100000
(200ms quota per 100ms period)"] Cgroup --> Kernel["CFS/EEVDF enforces it:
the group's threads are
throttled once they've
used 200ms of CPU time
within any 100ms period"] classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c class Cgroup info class Kernel warn ``` **What to notice:** a Kubernetes CPU limit of "2" doesn't mean "2 whole cores, always" in the way it's often described casually — it's implemented as a **quota within a period** (cgroup v2's `cpu.max`, expressed as ` ` in microseconds; the equivalent under the older cgroup v1 is two separate files, `cpu.cfs_quota_us` and `cpu.cfs_period_us`). A limit of 2 cores with the default 100ms period means the *combined* CPU time across all of that container's threads cannot exceed 200ms of core-time within any 100ms window — mechanically identical to 2 full cores' worth of time, but enforced in a way that has a genuinely important consequence covered in the next section. ```bash # Inspect a container's actual cgroup v2 CPU quota directly cat /sys/fs/cgroup/kubepods.slice/.../cpu.max # 200000 100000 # ^^^^^^ quota: 200ms of CPU time allowed... # ^^^^^^ ...within every 100ms period ``` > [!NOTE] > The period itself (100ms by default) is also configurable, though rarely changed in practice — a shorter period enforces the same overall ratio in smaller, more frequent slices (smoother, but less burst headroom per slice), while a longer period allows a bigger single burst before throttling, at the cost of coarser enforcement. Kubernetes doesn't expose this as a resource-spec field, so changing it requires kubelet-level configuration, which is why sizing the CPU limit itself is almost always the more practical lever a platform engineer actually reaches for. --- ## CPU Throttling — Why "Under the Limit" Still Stutters The quota-per-period mechanism from the previous section produces a genuinely counterintuitive real-world effect: a container can be **throttled** — hard-paused for the remainder of a period — even while its *average* CPU usage over a longer window looks comfortably under its limit. ```mermaid graph TD Burst["A multi-threaded checkout-service
pod briefly uses 4 cores' worth
of CPU simultaneously across
its threads, for 40ms"] --> Quota["cgroup quota: 200ms per
100ms period (2-core limit)"] Quota --> Consumed["4 cores x 40ms = 160ms
of quota consumed in
the FIRST 40ms of the period"] Consumed --> Remaining["Only 40ms of quota left
for the REMAINING 60ms
of this period"] Remaining --> Throttled["Threads are throttled —
paused — once quota
is exhausted, even
though 'average' usage
looks fine over a longer window"] classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c class Consumed warn class Throttled crit ``` **Why this happens even for a service that "isn't using much CPU":** a modern multi-threaded runtime (a language runtime with a thread pool, a garbage collector running on its own threads, a web server handling several requests concurrently) can genuinely burst well above its steady-state average for a short window — several threads all doing real work simultaneously for a handful of milliseconds — and the cgroup quota mechanism enforces its limit on that instantaneous, per-period burst, not on a smoothed long-term average. A container whose CPU usage graph looks like it never comes close to its limit, averaged over a minute, can still be spending a meaningful fraction of every 100ms period fully throttled. | Diagnostic | What it reveals | |---|---| | `cat /sys/fs/cgroup/.../cpu.stat` → `nr_throttled` / `throttled_time` | Directly counts how many periods a container has been throttled in, and total time spent throttled — the ground-truth signal | | Kubernetes `container_cpu_cfs_throttled_periods_total` (via cAdvisor/Prometheus) | The same signal, exposed as a standard cluster metric — a nonzero, climbing rate here is the real symptom, regardless of what average CPU utilization graphs show | | A latency graph with periodic, sub-second micro-stalls that don't correlate with average CPU utilization | The classic symptom pattern this section describes — worth checking throttling metrics specifically before assuming it's unrelated to CPU limits at all | **From the Trenches:** `checkout-service`'s p99 latency showed a consistent, small but real periodic stutter under moderate load, even though its Kubernetes CPU limit was set generously above its observed average utilization and the standard "CPU usage vs. limit" dashboard showed comfortable headroom the entire time. The immediate cause, found only after someone thought to check `container_cpu_cfs_throttled_periods_total` specifically, was a meaningful, steady rate of CFS throttling — the service's garbage-collector threads and request-handling threads were bursting together often enough, within individual 100ms windows, to exhaust the quota repeatedly, despite a comfortable-looking average. The underlying, two-levels-deep condition was that the team's capacity-planning process had only ever looked at average utilization against limits, because that was the metric the default dashboard surfaced prominently — throttling is a distinct metric that has to be deliberately added, and nobody had connected "our CPU limit looks generous" with "but is the workload's *burst* shape actually compatible with a 100ms quota window." The fix was raising the CPU limit specifically to give bursts more headroom (not because average usage demanded it) and adding the throttling metric to the team's standard dashboard for every CPU-limited service going forward. > [!TIP] > **Best practice:** never evaluate whether a Kubernetes CPU limit is "enough" using average utilization alone. Always check the container's throttling metric (`container_cpu_cfs_throttled_periods_total` or the equivalent `cpu.stat` fields) directly — a service can be throttled thousands of times a day while its average utilization graph looks perfectly healthy, and throttling is what actually produces user-facing latency micro-stalls. **Interview-ready line:** "A Kubernetes CPU limit is enforced as a hard quota within a fixed period, typically 100ms — not a smoothed average. A multi-threaded workload that briefly bursts across several threads at once can exhaust that quota and get throttled for the rest of the period, even while its longer-term average CPU usage looks comfortably under the limit. That's why throttling has to be checked as its own metric, not inferred from an average-utilization graph." --- ## CPU Requests vs. Limits — Shares vs. Hard Ceilings Kubernetes actually configures two related but mechanically distinct cgroup controls from one resource specification: **requests** set relative *shares* (how much of the contested CPU this container is entitled to versus its neighbors, when the node is busy), while **limits** set the hard quota-per-period ceiling from the previous two sections. ```mermaid graph TD Request["resources.requests.cpu: 500m"] --> Shares["cgroup cpu.weight —
a RELATIVE share used only
when the node is CPU-contended"] Limit["resources.limits.cpu: 2"] --> Quota["cgroup cpu.max —
an ABSOLUTE hard ceiling,
enforced every period,
contended or not"] classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c class Shares info class Quota warn ``` **What to notice:** a **request** never directly throttles anything by itself — it only determines this container's proportional share of the core(s) *when there's contention for them*, functioning much like the nice-value weighting from Section 6, but applied to a whole cgroup rather than one process. A **limit**, in contrast, is enforced unconditionally every single period, contended or not — a container can be hard-throttled by its limit even while running completely alone on an otherwise idle node, which is exactly the mechanism Section 9 described. | Kubernetes QoS class | How it's assigned | Scheduling consequence | |---|---|---| | `Guaranteed` | `requests == limits` for every container, for both CPU and memory | Qualifies for the Kubernetes CPU Manager's `static` policy (exclusive pinned cores — Part 1's Section 13) if the node supports it | | `Burstable` | `requests` set, but `limits` unset or higher than `requests` | Gets its requested share guaranteed under contention, can use more when available, subject to the limit's hard quota if one is set | | `BestEffort` | Neither `requests` nor `limits` set | Lowest scheduling priority, first to be throttled or evicted under node-level resource pressure | > [!IMPORTANT] > Setting a CPU **limit** is not a free safety net — it's a genuine trade-off. An unset limit lets a container use spare CPU capacity on an otherwise idle node freely, which is often exactly what's wanted for a bursty workload; a limit caps that upside in exchange for guaranteed predictability against noisy neighbors. Many real-world Kubernetes guidance discussions (worth cross-referencing with the [Kubernetes Deep Dive series' capacity planning chapter](/tutorials/kubernetes-deep-dive/16-capacity-planning)) recommend setting CPU **requests** carefully but leaving CPU **limits** unset for many workloads specifically to avoid Section 9's throttling behavior — this is a real, actively debated operational trade-off, not a universally settled default. **From the Trenches:** a platform team standardized on "always set both requests and limits, and always make them equal" as a blanket policy across every service, reasoning it gave the most predictable behavior everywhere. Several bursty, latency-tolerant batch-processing workloads saw their throughput drop noticeably after the policy rolled out, even though nothing about their actual resource needs had changed. The immediate cause was straightforward given this section's mechanics: forcing `requests == limits` (the `Guaranteed` QoS shape) on workloads that genuinely benefited from bursting into a node's spare capacity replaced their previous flexible ceiling with a hard one, throttling them at exactly their steady-state request value instead of letting them use idle capacity when available. The underlying, two-levels-deep condition was that the policy had been designed around the genuinely correct goal of predictability for latency-sensitive services, then applied uniformly to a workload category (bursty batch jobs) where predictability was never the actual priority — throughput was. The fix was scoping the `Guaranteed`-everywhere policy specifically to latency-sensitive services, while batch/background workloads kept `Burstable` requests-only configuration deliberately. --- ## Inside cpu.weight — How Kubernetes Requests Become a Real Scheduler Value Section 10 established that a Kubernetes CPU **request** becomes a relative *share* via cgroups, distinct from a limit's hard quota — this section makes that translation concrete, because the actual numbers involved are a frequent source of confusion when reading raw cgroup files directly during a deep investigation. Under cgroup v2, relative CPU share is expressed through the `cpu.weight` file, an integer from 1 to 10000 (the default is 100), applied proportionally: a cgroup with `cpu.weight` 200 gets roughly twice the CPU time of a sibling cgroup at `cpu.weight` 100, *when both are contending for the same cores*. Kubernetes computes this value from a pod's CPU request using a fixed conversion formula, roughly proportional to millicores requested. ```bash # A pod requesting 500m (0.5 CPU) — inspect the resulting cgroup weight directly cat /sys/fs/cgroup/kubepods.slice/.../cpu.weight # 51 <-- roughly proportional to the 500m request, NOT a literal core count # Compare against a neighbor requesting 2 full CPUs cat /sys/fs/cgroup/kubepods.slice/.../cpu.weight # 204 <-- roughly 4x the first pod's weight, matching its 4x larger request ``` | Field | What it controls | Scope | |---|---|---| | `cpu.weight` | Relative share under contention (derived from CPU **requests**) | Only matters when multiple cgroups are actively competing for the same cores | | `cpu.max` | Absolute hard quota-per-period ceiling (derived from CPU **limits**) | Enforced unconditionally, every period, contended or not (Section 9) | **What this resolves in practice:** two pods with identical CPU limits but very different CPU requests will behave identically when the node is idle (nothing to contend over, and limits don't care about weight) but diverge sharply the moment the node becomes CPU-contended — the higher-request pod gets a proportionally larger share of whatever CPU time is actually available, exactly matching the `Guaranteed`/`Burstable`/`BestEffort` QoS distinctions from Section 10, now traceable to one concrete, inspectable file. > [!NOTE] > `cpu.weight` is the cgroup v2 mechanism; the equivalent under the older cgroup v1 hierarchy is `cpu.shares`, using a different numeric range (default 1024) but the same proportional-share concept. Most current Kubernetes clusters run on cgroup v2 by default, but recognizing both names avoids confusion when reading older documentation or an unusually configured host. **From the Trenches:** during a noisy-neighbor investigation, a team compared two `catalog-service` pods' `cpu.weight` values directly and found one at `51` and the other at `1000`, and initially assumed the second pod had somehow been misconfigured with a 20x larger CPU request by mistake. The immediate cause, on checking the actual pod specs, was that the second pod's CPU **request** genuinely was set roughly 20x higher — its weight value was correctly derived, not a bug. The underlying, two-levels-deep condition was that nobody investigating had actually looked at the pod spec first, jumping straight to the unfamiliar-looking raw cgroup number and assuming *it* was the anomaly rather than checking the higher-level Kubernetes resource configuration that produced it. The lesson generalizes: `cpu.weight` is a derived value, not a source of truth — any investigation into an unexpected weight should start from the pod's actual `resources.requests.cpu`, not from the cgroup file itself. --- ## cgroup v1 vs. v2 — Why the File Paths and Semantics Differ Every cgroup example so far in this chapter has used cgroup **v2** syntax (`cpu.max`, `cpu.weight`) — but a meaningful number of hosts, especially older ones or specific managed-Kubernetes node images, still run cgroup **v1**, which uses different file names, a different hierarchy structure, and slightly different semantics. Recognizing both is worth having, since encountering v1 syntax mid-investigation without knowing it's a version difference (not a typo or a different mechanism entirely) can cost real debugging time. | Concept | cgroup v1 | cgroup v2 | |---|---|---| | Hierarchy structure | Separate, independent hierarchies per controller (a process could be in different groups for CPU vs. memory vs. I/O) | One single unified hierarchy — a process belongs to exactly one cgroup, which controls all resource types together | | CPU hard quota | Two separate files: `cpu.cfs_quota_us` and `cpu.cfs_period_us` | One combined file: `cpu.max`, formatted ` ` | | CPU relative share | `cpu.shares` (default 1024, unbounded range) | `cpu.weight` (default 100, range 1-10000) | | Throttling stats | `cpu.stat` (present, but with a narrower field set than v2) | `cpu.stat` (same filename, richer field set — `nr_periods`, `nr_throttled`, `throttled_usec`) | | Current adoption | Legacy — present on older kernels/distributions and some unmigrated hosts | The default on current Kubernetes versions and current major Linux distributions | ```bash # Confirm which cgroup version a host is actually using before trusting either # file layout — the presence of this single file is the definitive signal mount | grep cgroup2 # cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime) # ^^^^^^^ if this line exists, the host is on cgroup v2 (or a hybrid setup) ``` > [!NOTE] > The underlying kernel scheduling mechanism this whole chapter describes (CFS/EEVDF, vruntime, quota-per-period enforcement) is identical either way — cgroup v1 vs. v2 changes only the *control-file interface* used to configure it, not the scheduling algorithm itself. A container's actual runtime CPU behavior is the same under either version given equivalent configuration; only the paths and field names used to inspect or set that configuration differ. Some hosts run a genuine **hybrid** configuration during migration — cgroup v1 controllers still mounted for some subsystems while v2 handles others — which is worth knowing exists specifically so an unexpected mix of v1 and v2 file paths on the same host isn't mistaken for a misconfiguration. `mount | grep cgroup` (without the `2`) shows every mounted cgroup filesystem, v1 and v2 alike, and is the more complete diagnostic when a host's cgroup layout looks inconsistent with what either pure version would produce on its own. --- ## Multi-Threaded Runtimes and the Scheduler — Why Language Choice Interacts With Throttling Section 9's CPU throttling problem has a distinct, extremely common real-world trigger worth naming directly: a language runtime that doesn't know how many CPUs its own container is actually allowed to use, and spawns far more OS threads than the cgroup quota can support without constant throttling. The Go runtime is the canonical example, specifically because its historical default made the problem almost guaranteed rather than occasional. `GOMAXPROCS` — the number of OS threads the Go runtime will use to run goroutines in parallel — historically defaulted to the **host's total logical CPU count**, discovered from `/proc/cpuinfo`, with no awareness of any cgroup CPU limit constraining the actual container. A Go service on a 64-core host, running in a container limited to 2 CPUs, would default to `GOMAXPROCS=64` — scheduling far more parallel work than a 2-CPU quota could sustain, guaranteeing frequent, severe throttling under any real concurrent load. ```mermaid graph TD Host["64-core physical host"] --> Container["Container: cgroup CPU limit = 2"] Container --> GoRuntime["Go runtime (pre-1.25 default):
GOMAXPROCS = 64
(reads host CPU count, NOT the cgroup limit)"] GoRuntime --> Threads["Schedules up to 64 OS threads
worth of parallel goroutine work"] Threads --> Throttle["Cgroup quota (Section 9) is
a small fraction of what the
runtime is actually trying to use —
severe, near-constant throttling"] classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c class GoRuntime warn class Throttle crit ``` | Approach | Mechanism | Status | |---|---|---| | `go.uber.org/automaxprocs` library | Reads the cgroup CPU quota directly at startup and calls `runtime.GOMAXPROCS()` to correct the default | A widely adopted, explicit workaround, still relevant for any Go version before 1.25 | | Go 1.25+ (container-aware default) | The runtime itself reads cgroup v2 CPU bandwidth limits and sets `GOMAXPROCS` correctly by default, re-checking periodically if the limit changes | The underlying problem is fixed at the language level going forward — worth confirming which Go version a service actually runs before assuming this class of throttling is ruled out | | Manually setting `GOMAXPROCS` via environment variable | Explicit, static override matching the known container CPU limit | Works, but silently wrong if the limit is ever changed without updating the corresponding environment variable | This isn't a Go-specific quirk in principle — any runtime that sizes its own internal thread pool from host-level CPU discovery rather than the container's actual cgroup constraints is vulnerable to the identical failure mode. JVM-based services have analogous container-awareness settings (modern JVMs read cgroup limits by default in current versions, but this wasn't always true either), and any hand-rolled thread-pool sizing logic that calls a host-CPU-count API without cross-checking the cgroup limit reproduces the same bug from first principles. **From the Trenches:** a newly containerized Go service for `catalog-service`'s search-indexing pipeline showed severe, consistent throttling in production despite a CPU limit that had been sized generously relative to the service's expected average load, based on load-testing results from a bare-metal development machine with far fewer cores than the production host. The immediate cause was exactly this section's failure mode: the Go runtime, running on the physical host's actual (high) core count with no cgroup awareness, had set `GOMAXPROCS` far above the container's actual 2-CPU limit, scheduling enough parallel goroutine work to exhaust the quota within milliseconds of every 100ms period. The underlying, two-levels-deep condition was that the development machine's much lower core count had coincidentally kept `GOMAXPROCS` in a range that never triggered severe throttling during testing — the bug was real throughout, but its symptom only became severe once the *production* host's much higher core count widened the gap between "what the runtime thinks it can use" and "what the cgroup actually allows." The fix was adopting `go.uber.org/automaxprocs`, which resolved the throttling immediately without any application-code change. > [!TIP] > **Best practice:** for any service written in a language whose runtime sizes internal parallelism from host CPU discovery, explicitly confirm cgroup-awareness before deploying into a resource-limited container — check the runtime/library version against known container-aware fixes (Go 1.25+, current JVM defaults), or add an explicit correction library, rather than assuming a generously-sized CPU limit alone prevents this specific throttling pattern. --- ## Wait Queues and Wakeups — How a Sleeping Thread Becomes Runnable Again Section 2 described a thread moving into sleep states (`S`/`D`) when it blocks, and back to runnable "when the event it was waiting for occurs" — this section makes that mechanism concrete, connecting directly back to Part 1's interrupt coverage. ```mermaid sequenceDiagram participant App as checkout-service thread participant WQ as Kernel wait queue participant Disk as Disk controller participant Sched as Scheduler App->>WQ: Blocks on a disk read, added to a wait queue, state -> D Note over App: Off the run queue entirely —
not competing for CPU at all Disk->>Disk: Read completes Disk->>Sched: Hardware interrupt fires (Part 1's interrupt mechanism) Sched->>WQ: Interrupt handler calls wake_up() on the wait queue WQ->>App: Thread's state changes back to Runnable App->>Sched: Now competes normally for a core (this chapter's CFS/EEVDF fairness) ``` **What to notice:** a blocked thread isn't being "checked on" repeatedly by the scheduler while it waits — it's parked on a **wait queue** specific to whatever it's blocked on (a particular disk operation, a particular socket, a particular lock), entirely off any run queue, consuming no scheduling overhead at all until the specific event it's waiting for calls `wake_up()` on that exact queue. This is precisely why a host can have thousands of blocked threads with no scheduling cost, but why a bug that fails to call the wake-up path correctly (a lost wakeup, a specific class of kernel/driver bug) can leave a thread parked in sleep state indefinitely, waiting for an event that will now never arrive to un-park it. | Blocking event | Wait queue triggered by | Resulting state on block | |---|---|---| | Disk/NFS I/O | The storage driver's completion interrupt (Part 1's interrupt handling) | `D` — uninterruptible, can't be signaled away | | Socket read with no data yet | The network stack's packet-arrival path (also interrupt-driven, per Part 1) | `S` — interruptible, can be woken by a signal too | | Waiting on a mutex/lock | The lock's own release path, called by whichever thread currently holds it | `S`, typically | | A timer (`sleep()`, a scheduled task) | The kernel's timer-expiry mechanism, itself ultimately interrupt-driven | `S` | > [!NOTE] > This is the precise mechanical link between Part 1's hardware-interrupt coverage and this chapter's scheduling machinery: an interrupt handler is very often *itself* the thing that calls `wake_up()` on a wait queue, moving a blocked thread back to runnable so the scheduler can consider it again. Interrupts and scheduling aren't two unrelated topics — a huge fraction of real-world scheduling activity is directly triggered by hardware events arriving via the interrupt path this series' first chapter covered. ```bash # Watch a specific process's voluntary sleep/wake cycles directly cat /proc//wchan # ep_poll <-- currently parked in the epoll wait path, waiting on socket activity # A quick way to see which syscall a blocked thread is actually waiting inside cat /proc//stack 2>/dev/null | head -3 # [<0>] ep_poll+0x2a3/0x3c0 # [<0>] do_epoll_wait+0xb0/0xd0 # [<0>] __x64_sys_epoll_wait+0x63/0xa0 ``` **What to notice:** `wchan` (wait channel) names the exact kernel function a blocked thread is parked inside — here, `ep_poll`, confirming the thread is genuinely waiting on `epoll` (socket readiness), not stuck or leaked. This turns "why is this thread not running" from a guess into a direct, one-command answer. --- ## Priority Inversion — When a Low-Priority Thread Blocks a High-Priority One Every mechanism covered so far assumes priority works the way it sounds like it should: a higher-priority thread gets the CPU sooner. **Priority inversion** is the specific, classic scenario where that assumption breaks down — not because the scheduler is wrong, but because scheduling and locking are two separate mechanisms that can interact in a genuinely counterintuitive way. ```mermaid sequenceDiagram participant Low as Low-priority thread participant Mid as Medium-priority thread participant High as High-priority thread participant Lock as Shared lock Low->>Lock: Acquires the lock first High->>Lock: Needs the same lock — BLOCKS, waiting Mid->>Mid: Becomes runnable, has no interest in the lock Note over Mid: Scheduler correctly prefers Mid
over Low (Mid has higher priority) Mid->>Mid: Runs, preempting Low Note over High: High-priority thread is now blocked
indirectly by a MEDIUM-priority thread
it has no direct relationship with at all Low->>Lock: Eventually resumes, releases lock High->>Lock: Finally acquires it and proceeds ``` **What to notice:** the high-priority thread isn't blocked by the low-priority thread directly holding the lock (that part is expected and bounded) — it's blocked *indirectly and unboundedly* by the medium-priority thread, which has no interest in the lock at all but keeps preempting the low-priority thread that's holding it, preventing the low-priority thread from ever finishing and releasing it. This is exactly the failure mode that famously caused the Mars Pathfinder mission's system resets in 1997 — a real, historically significant example, not a theoretical curiosity. The standard fix is **priority inheritance**: when a high-priority thread blocks on a lock held by a lower-priority one, the lock-holder is temporarily boosted to the waiter's priority for as long as it holds the lock, specifically so a medium-priority thread can no longer preempt it and indirectly stall the high-priority waiter. Linux's `pthread` mutexes support priority inheritance explicitly (`PTHREAD_PRIO_INHERIT`), though it isn't the default and has to be deliberately requested. > [!IMPORTANT] > Priority inversion matters most for the same narrow category of genuinely latency-critical, priority-differentiated workloads this chapter has flagged before (real-time scheduling classes, `PREEMPT_RT` kernels) — in a typical web-service fleet where most application threads run at the same default priority (nice 0, `SCHED_NORMAL`), there's no meaningful priority difference for inversion to exploit in the first place. It's genuinely important to recognize by name for an interview or a real-time/embedded context, and largely a non-issue for ordinary containerized service scheduling. --- ## Load Average — What That Number Actually Means `uptime`'s three numbers — the 1-, 5-, and 15-minute **load average** — are among the most-viewed and most-misunderstood figures in all of systems operations, largely because "load" sounds like it should mean "CPU utilization," and on Linux specifically, it does not. ```mermaid graph LR R["Runnable threads
(waiting for or using a CPU)"] --> LA["Linux Load Average"] D["Threads in UNINTERRUPTIBLE
sleep (D state — disk/NFS I/O)"] --> LA S["Threads in interruptible
sleep (S state)"] -.->|"NOT counted"| LA classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c classDef muted fill:#eaeef1,stroke:#c3ccd4,color:#10161c class R ok class D crit class S muted ``` **What to notice:** Linux's load average counts both runnable threads *and* threads stuck in uninterruptible sleep (`D` state, Section 2) — a decision made decades ago specifically because a system full of processes blocked on slow disk I/O is genuinely "loaded" in every practical sense, even though no CPU cycles are being consumed. This means a load average spike with low CPU utilization is not a contradiction or a bug — it's frequently the single clearest signal of a storage/I/O bottleneck (fully covered in Part 4), not a CPU one, and is one of the most common causes of "the load average looks scary but CPU utilization looks fine" confusion. | Load average reading | Correct interpretation on an 8-core host | |---|---| | 4.0 | Roughly half the host's total core capacity is in demand, on average, over that window — not necessarily a problem | | 8.0 | Demand roughly matches total core capacity — the host is fully utilized, at the edge of a legitimate reason for concern | | 16.0, but CPU utilization (`%us`+`%sy`) is low | A strong signal that most of that "load" is threads in `D` state waiting on I/O, not CPU contention — check `iostat`/disk latency next, not CPU capacity | | Rising 1-minute figure, flat 15-minute figure | A recent, possibly still-developing spike — worth watching, not yet a confirmed trend | **From the Trenches:** an alert fired for `inventory-service`'s load average crossing 3x its core count, and the on-call engineer's first instinct was to scale the deployment horizontally, following a runbook written for CPU-saturation incidents. `mpstat` showed CPU utilization barely above 20%. The immediate cause, found by checking process states directly (`ps aux | grep " D "`), was a large number of threads stuck in uninterruptible sleep waiting on a specific slow, degraded EBS volume backing the service's local disk cache — genuine I/O saturation, not CPU demand at all. The underlying, two-levels-deep condition was that the on-call runbook had been written assuming "high load average" always meant "needs more compute," a reasonable-sounding but incomplete generalization that had never been corrected because most prior incidents genuinely had been CPU-bound — nobody had encountered, and therefore documented, the I/O-bound case this specific incident represented. Scaling horizontally would have added more instances all still depending on the same degraded volume, doing nothing to fix the actual bottleneck. The fix was addressing the underlying storage issue (covered fully in Part 4) and rewriting the runbook to check the CPU-utilization-vs-load-average relationship from this section as the very first triage step. --- ## Load Balancing Across Cores Every section so far has implicitly described scheduling as if there were one single run queue — in reality, modern Linux maintains a **separate run queue per core**, for a direct, important reason established in Part 1: moving a thread between cores costs a cache-locality penalty (a "cold" cache and, on multi-socket hosts, a possible NUMA penalty), so the scheduler is deliberately biased toward keeping a thread on the same core it already ran on, rather than constantly redistributing for perfect instantaneous fairness across the whole machine. ```mermaid graph TD subgraph SMT["SMT domain
(logical threads on ONE core)"] T0["Thread 0"] -.->|"cheapest to balance:
same core, shared cache"| T1["Thread 1"] end subgraph SMP["SMP domain
(cores on one socket)"] C0["Core 0"] -.->|"moderate cost:
shared L3, same NUMA node"| C1["Core 1"] end subgraph NUMA["NUMA domain
(across sockets)"] S0["Socket 0"] -.->|"most expensive:
cross-socket, remote memory"| S1["Socket 1"] end classDef ok fill:#e5f5ea,stroke:#1f8a4c,color:#10161c classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c class SMT ok class SMP warn class NUMA crit ``` **What to notice:** the kernel organizes cores into a hierarchy of **scheduling domains** (SMT siblings, then cores on the same socket, then across sockets/NUMA nodes) that directly mirrors Part 1's hardware topology, and load balancing is deliberately reluctant to move work across a more expensive domain boundary unless the imbalance is significant enough to justify the migration cost. This is exactly why a busy 16-core host can sometimes show noticeably uneven per-core utilization in `mpstat -P ALL` without that being a bug — the scheduler is weighing "perfectly even utilization right now" against "avoid an expensive cross-NUMA migration for a transient imbalance," and deliberately chooses some short-term unevenness over constant thrashing. The trigger for a rebalancing pass is itself tied to this chapter's scheduler tick (covered later in this chapter): on each tick, the kernel checks whether the current core's run queue is significantly busier than others in its scheduling domain, and only initiates a migration when the imbalance clears a threshold — this periodic, threshold-gated design is itself a deliberate trade-off against constantly re-evaluating placement on every single scheduling decision, which would add overhead disproportionate to the benefit for most workloads. **From the Trenches:** a team noticed `mpstat -P ALL` consistently showing cores 0-7 more heavily utilized than cores 8-15 on a dual-socket `inventory-service` database host, and assumed the scheduler was malfunctioning or misconfigured. The immediate cause, on closer investigation, was correct scheduler behavior, not a bug: the database's connection-handling threads had been created early in the process's life and had settled onto cores 0-7's NUMA node (where their memory was also allocated), and ordinary load-balancing pressure hadn't been strong enough to justify the cross-NUMA migration cost of moving some of them to the emptier node, since doing so would have traded a load-balance improvement for a NUMA-locality penalty on every future memory access. The underlying, two-levels-deep condition was that the team's mental model of "the scheduler should spread load evenly across every core" was accurate for CFS/EEVDF's *local* fairness guarantee but ignored the deliberate cost-aware trade-off scheduling domains introduce specifically to avoid Part 1's NUMA penalty — an imbalance the scheduler was choosing on purpose, not failing to notice. The fix wasn't a scheduler tweak at all; it was explicitly NUMA-aware capacity planning (sizing the workload to fit comfortably within one node, per Part 1's own NUMA section) rather than expecting the scheduler to override its own locality-preserving design. > [!TIP] > **Best practice:** don't interpret uneven per-core utilization alone as a scheduler problem. Cross-reference it against `numactl --hardware` (Part 1) first — uneven utilization that aligns with NUMA node boundaries is very often the scheduler correctly avoiding an expensive migration, not a sign that something needs fixing. --- ## Preemption and Scheduling Latency **Scheduling latency** — the time between a thread becoming runnable and actually getting a core — is a distinct, separately-important measurement from the throughput-focused fairness this chapter has covered so far. A thread can be perfectly "fairly" scheduled over the long run and still experience occasional, real latency spikes waiting for its turn. **Preemption** is what keeps this latency bounded: even a currently-running thread can be interrupted mid-execution if a higher-priority thread becomes runnable (Section 7's real-time classes) or, under CFS/EEVDF, if fairness accounting determines another thread has gone too long without a turn. The kernel's preemption model is itself configurable at build time (`CONFIG_PREEMPT`), trading maximum throughput against worst-case scheduling latency — a general-purpose server kernel typically favors throughput with occasional longer latency tails, while specialized low-latency kernel configurations (`PREEMPT_RT`) sacrifice some throughput for much tighter, more predictable worst-case latency guarantees. > [!TIP] > **Best practice:** for the small subset of workloads where worst-case scheduling latency genuinely matters more than aggregate throughput (the same latency-critical category Part 1's CPU-pinning section addressed from a different angle), check whether the host is running a `PREEMPT_RT`-patched or standard kernel (`uname -a` reports this) before assuming a scheduling-latency problem can be fixed purely through application-level tuning — some worst-case latency floors are set by the kernel's own preemption model, not by anything configurable from inside a container. **Interview-ready line:** "Scheduling latency and throughput are genuinely different concerns — a fair-share scheduler can deliver excellent long-run throughput to every thread while still letting an individual thread wait longer than ideal for its next turn. Preemption bounds that wait by letting a higher-priority or overdue thread interrupt whatever's currently running; how aggressively the kernel does this is itself a configurable trade-off, from general-purpose `CONFIG_PREEMPT` defaults through to `PREEMPT_RT` for workloads that need a much tighter worst-case guarantee." --- ## The Scheduler Tick — How Often the Kernel Even Gets a Chance to Reschedule Everything covered so far assumes the scheduler is invoked whenever it needs to be — but the kernel actually gets most of its opportunities to reschedule from a **timer interrupt** (Part 1's Section 7 covers interrupts generally), firing at a configurable rate called `CONFIG_HZ`, historically 100, 250, or 1000 times per second depending on the kernel build. Each tick gives the scheduler a chance to check whether the currently running thread's time is up and a switch is warranted. | `CONFIG_HZ` | Tick interval | Trade-off | |---|---|---| | 100 Hz | 10ms | Lower timer-interrupt overhead, coarser scheduling granularity (worse worst-case latency) | | 250 Hz | 4ms | A common general-purpose server default, balancing overhead against responsiveness | | 1000 Hz | 1ms | Finer scheduling granularity and lower worst-case latency, at the cost of more frequent timer-interrupt overhead | Modern Linux also supports **tickless operation** (`NO_HZ` / `NO_HZ_FULL`), which suppresses the periodic timer interrupt entirely on cores that have nothing to schedule (an idle core doesn't need to be woken up every tick just to confirm it's still idle — this directly connects to Part 1's C-state coverage, since an unnecessary periodic wake-up would defeat the purpose of a deep idle state) or, with `NO_HZ_FULL`, even on a core running exactly one runnable thread with nothing else contending for it, letting that thread run completely uninterrupted by scheduling-tick overhead — a specialized configuration used in latency-critical, dedicated-core scenarios (high-frequency trading, some telecom/5G workloads) rather than general-purpose server fleets. > [!NOTE] > This is genuinely low-level kernel-tuning territory that the overwhelming majority of platform engineers will never need to touch directly — cloud provider kernels and standard Linux distributions ship sensible defaults. It's worth knowing by name specifically because it explains *why* scheduling decisions aren't instantaneous even for CFS/EEVDF's fairness math: the scheduler can only act as often as it's actually invoked, and the tick rate sets a real, if usually invisible, floor on how quickly it can react. --- ## Voluntary vs. Involuntary Context Switches — What Each One Tells You Part 1 established that context switches carry a real performance cost; this chapter's scheduling mechanisms make it possible to distinguish *why* a given context switch happened, which turns a single aggregate `cs` number (Part 1's `vmstat` column) into a genuinely diagnostic signal. | Type | What triggers it | What a high rate tells you | |---|---|---| | **Voluntary** | The thread itself blocks — on I/O, a lock, a syscall — and yields the core because it has nothing to do until that resolves | Expected and healthy for I/O-bound work (Part 1's classification); a rising voluntary rate on a CPU-bound workload can indicate growing lock contention | | **Involuntary** | The thread was preempted while still runnable and willing to keep executing — the scheduler decided something else deserved the core instead | A high rate relative to voluntary switches is a strong, specific signal of CPU contention — more runnable threads competing for cores than the host can serve without frequent interruption | **Where to find each count directly**, per process: `/proc//status`'s `voluntary_ctxt_switches` and `nonvoluntary_ctxt_switches` fields (Section 22's diagnostic block shows the exact command). Tracking their *ratio* over time, not just their raw values, is what actually reveals a trend — a service that always runs at roughly 90% voluntary is behaving normally even if its absolute switch count grows with traffic; a service whose involuntary share climbs from 5% to 40% over a month is the pattern actually worth alerting on. **What to notice:** a service that's genuinely CPU-bound and healthy should show mostly *voluntary* switches if it's well below saturation, or a climbing share of *involuntary* switches specifically as it approaches or exceeds the host's real capacity — the ratio between the two, tracked over time, is a more specific early-warning signal than the aggregate context-switch rate alone, because it distinguishes "this thread chose to yield" from "this thread was forced off the core against its will." **From the Trenches:** a capacity-planning review flagged `checkout-service`'s steadily climbing `nonvoluntary_ctxt_switches` count as a concerning trend, while the aggregate context-switch rate from `vmstat` had looked stable for weeks. The immediate cause was that the service's thread pool had been sized correctly at deployment time, but organic request-volume growth over several months had gradually pushed the host closer to genuine CPU saturation — the *aggregate* switch rate stayed misleadingly flat because voluntary switches (I/O waits) were actually decreasing as CPU contention grew (threads spent less time blocked and more time runnable-but-waiting), masking the shift in a metric that only looked at the total. The underlying, two-levels-deep condition was that the team's capacity dashboard tracked the total context-switch rate as a single number, which happened to stay flat purely because two opposite trends (falling voluntary, rising involuntary switches) were canceling out in the aggregate — a genuinely misleading coincidence that a breakdown by type would have caught months earlier. The fix was splitting the dashboard metric into voluntary and involuntary components specifically, catching this exact class of gradual capacity erosion before it became a user-facing incident. > [!TIP] > **Best practice:** track voluntary and involuntary context-switch rates as separate metrics, not just their sum. A rising involuntary rate — even with a flat or falling aggregate — is one of the earliest, cheapest-to-collect leading indicators of approaching CPU saturation available on any Linux host. --- ## How CPU Affinity Constrains the Scheduler's Choices Part 1 introduced CPU affinity and pinning from the hardware-locality angle — keeping a process's cache and NUMA locality consistent by restricting it to specific cores. This chapter's scheduling machinery is what actually *enforces* that restriction moment to moment, and understanding the interaction resolves a subtlety worth being explicit about: pinning doesn't replace CFS/EEVDF, it narrows the set of cores those algorithms are allowed to choose from. ```mermaid graph TD Thread["A thread with CPU affinity
set to cores 4-7 (Part 1)"] --> Eligible["Scheduler's eligible core set
for this thread: ONLY 4-7"] Eligible --> Fairness["Within that restricted set,
CFS/EEVDF fairness math
(vruntime, nice, cgroup weight)
still applies normally"] Fairness --> Pick["Scheduler picks among
cores 4-7 exactly as it would
among all 16 cores otherwise —
just with a smaller menu"] classDef info fill:#e5f0fa,stroke:#1d6fb8,color:#10161c classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c class Eligible warn class Fairness info ``` **What to notice:** pinning a thread to a set of cores doesn't grant it exclusive access to them unless it's *also* configured through the Kubernetes CPU Manager `static` policy or an equivalent exclusive-reservation mechanism (Part 1's distinction) — a thread merely restricted to cores 4-7 via plain affinity still competes for fair-share time on those specific cores against anything else also scheduled there, using exactly the same vruntime/priority math covered throughout this chapter. Exclusive pinning is a *stronger*, separate guarantee layered on top of affinity, not a different mechanism entirely. | Configuration | What the scheduler actually does | |---|---| | No affinity set | Free to run the thread on any core, balanced across scheduling domains (Section 17) | | Affinity restricted to a subset of cores, no exclusivity | Fairness math applies normally, but only among threads sharing that same restricted core set | | Affinity restricted + Kubernetes CPU Manager `static` policy (exclusive) | Those specific cores are removed from the general scheduling pool entirely for other workloads — genuinely dedicated, not just preferred | > [!NOTE] > This is exactly why Part 1's warning against over-applying pinning matters mechanically, not just as general advice: restricting many workloads to small, overlapping core subsets without exclusivity doesn't reduce contention at all — it just concentrates the same fairness competition this chapter describes onto a smaller run-queue, which can make contention *more* visible, not less, if the restricted set is too small for the combined demand placed on it. --- ## Reading Scheduler Behavior on a Real Machine Every mechanism in this chapter is directly observable on a real host, extending Part 1's diagnostic toolkit with scheduler-specific commands. ```bash # Per-process CPU scheduling stats, including voluntary/involuntary context switches cat /proc//status | grep -i ctxt # voluntary_ctxt_switches: 1842 # nonvoluntary_ctxt_switches: 93 <-- high relative to voluntary = contention signal # Live per-process CPU accounting, refreshed like top but scriptable pidstat -p 1 # Time UID PID %usr %system %guest %wait %CPU CPU Command # 10:15:01 1000 4821 45.0 12.0 0.0 8.0 57.0 3 checkout-svc # A container's cgroup-level throttling counters, directly from cgroup v2 cat /sys/fs/cgroup/kubepods.slice/.../cpu.stat # usage_usec 182340021 # nr_periods 91200 # nr_throttled 412 <-- nonzero and climbing is the Section 9 signal # throttled_usec 8391044 # Confirm the effective nice value and scheduling policy a process is actually running under chrt -p # pid 4821's current scheduling policy: SCHED_OTHER # pid 4821's current scheduling priority: 0 ps -o pid,ni,cls,comm -p # PID NI CLS COMMAND # 4821 10 TS checkout-svc <-- NI=10: niced down; CLS=TS = SCHED_OTHER (CFS/EEVDF) ``` Between Part 1's topology tools and this chapter's scheduler-specific ones, a platform engineer can now answer not just "how much CPU capacity does this host have," but "is the scheduler actually delivering that capacity to the threads that need it, fairly and with acceptable latency" — the question that actually determines real-world service performance. **A realistic worked triage, tying the chapter together:** `checkout-service`'s p99 latency alert fires. The investigation sequence a platform engineer familiar with this chapter would actually run, in order: 1. Check `container_cpu_cfs_throttled_periods_total` first (Section 9) — if it's climbing, the investigation is very likely over: raise the limit, or investigate why the workload bursts harder than expected (Section 13's runtime-awareness angle if the service is Go, JVM, or similar). 2. If throttling is flat, check `mpstat`'s load average against actual CPU utilization (Section 16, and Part 1's category breakdown) — a high load average with low CPU utilization redirects the investigation toward I/O (Part 4), not scheduling at all. 3. If neither shows an obvious signal, break down context switches by type (Section 20) — a rising involuntary rate with a flat aggregate is the subtle, easy-to-miss capacity-erosion signal from this chapter's own From the Trenches example. 4. Only after all three come back clean is it reasonable to look outside this chapter's scope entirely — network (this site's Networking Deep Dive series), application-level locking, or a genuine code-level regression. This ordering isn't arbitrary — it's cheapest-and-most-likely-first: throttling is a single metric to check and explains the single most common class of Kubernetes CPU-related latency complaint, which is exactly why it belongs at the top of the list rather than being discovered after hours spent elsewhere. --- ## Garbage Collection as a Scheduling Event A managed-runtime service (JVM, Go, .NET, and most others with automatic memory management) periodically runs a **garbage collector** — and from this chapter's perspective, a GC cycle is fundamentally a scheduling event: it consumes real CPU time, competes with request-handling threads for cores, and in some collector designs, briefly stops every other thread in the process entirely. ```mermaid graph TD App["Application threads
handling requests"] --> GCTrigger{"GC threshold reached"} GCTrigger -->|"Stop-the-world collector
(older/simpler designs)"| STW["ALL application threads
PAUSED for the collection —
a direct, visible latency spike"] GCTrigger -->|"Concurrent collector
(most modern runtimes)"| Concurrent["GC runs on its OWN threads,
CONCURRENTLY with app threads —
competes for CPU time via
this chapter's normal scheduling,
rather than pausing everything"] classDef crit fill:#fbe8e6,stroke:#b3261e,color:#10161c classDef warn fill:#fbeee0,stroke:#b8650f,color:#10161c class STW crit class Concurrent warn ``` **What to notice:** even a "concurrent" collector that doesn't stop every thread still directly consumes cgroup CPU quota (this chapter's earlier CPU Throttling coverage) — GC threads running alongside request-handling threads are exactly the kind of multi-threaded burst behavior that can push a container over its quota within a single period, producing throttling that has nothing to do with the application's own request-handling code at all. A container CPU-limited too tightly for its collector's real behavior can show throttling that scales with *allocation rate* (how fast the application creates garbage), not with request volume — a genuinely confusing pattern if the connection to GC isn't already suspected. **From the Trenches:** `catalog-service`'s JVM-based search-ranking component showed periodic latency spikes that correlated with neither request volume nor any application-level event teams could find, until someone cross-referenced the timing against the JVM's own GC logs and found an exact match — every spike lined up with a garbage-collection cycle. The immediate cause was a stop-the-world young-generation collection running long enough, under the service's specific allocation pattern (creating many short-lived objects per request), to produce a real, user-visible pause. The underlying, two-levels-deep condition was that the service's CPU limit had been sized against its request-handling logic's steady-state needs, with no separate accounting for GC's own periodic CPU demand — the collector's threads were real, scheduled competitors for the same quota this chapter's throttling mechanism enforces, and nobody had modeled that competition into the original sizing. The fix combined two changes: tuning the JVM's collector settings to reduce individual pause duration, and increasing the CPU limit specifically to give collection cycles headroom without starving concurrent request handling. > [!TIP] > **Best practice:** for any managed-runtime service, correlate latency-spike timing against the runtime's own GC logs before assuming an application-level cause. This is a cheap, high-signal check — most managed runtimes can log GC events with timestamps directly comparable to a latency dashboard's spike timing — and it directly connects to sizing CPU limits correctly, since GC is real, quota-consuming scheduled work like everything else in this chapter. --- ## How These Concepts Show Up on the Cloud Bill Following Part 1's own closing pattern, this is a deliberate synthesis, not a new topic — connecting this chapter's scheduling mechanisms back to a cost conversation most platform teams have every quarter. | Chapter concept | Cost implication | The FinOps-relevant question to ask | |---|---|---| | CPU requests (relative shares) | Under-requesting to "save cost" degrades a service's real share the moment the node gets busy | Are requests set to reflect genuine steady-state need, not an artificially low number chosen to fit more pods per node? | | CPU limits and throttling | An overly tight limit throttles a workload that would otherwise fit comfortably in its actual usage pattern, forcing wasteful horizontal scale-out to compensate | Has the limit been sized against real burst behavior, or just a comfortable-looking average utilization number? | | Container-unaware runtime thread pools | A Go/JVM service defaulting its thread pool to host core count in a small container wastes the throttled cycles it never should have scheduled in the first place | Does the runtime version or configuration actually respect the container's cgroup CPU limit? | | CPU pinning / exclusive cores | Every exclusively reserved core is removed from the shared pool, directly reducing how many total workloads fit on that node | Is the tail-latency benefit of exclusive pinning worth its real node-density cost for this specific workload? | | Nice values and priority tuning | A correctly deprioritized background job avoids needing dedicated, separately-paid-for capacity just to keep it from disturbing foreground services | Could soft deprioritization (nice, cgroup weight) achieve the same isolation a more expensive dedicated node pool is currently providing? | | GC-driven CPU demand | Sizing purely for request-handling load without accounting for collector overhead leads to either unexplained throttling or an over-provisioned limit padded "just in case" | Has the collector's own CPU footprint been measured and included in capacity sizing, rather than folded into a vague safety margin? | | Burstable vs. fixed-performance node pools (Part 1) | Scheduling-aware workloads (bursty, tolerant of throttling) fit burstable economics; latency-critical ones don't | Does the workload's scheduling profile from this chapter actually match the instance family's underlying cost model? | > [!TIP] > **Best practice:** treat CPU request/limit sizing as a recurring review, not a one-time deployment decision — a request or limit that was correct at launch drifts out of alignment as real traffic patterns, code paths, and runtime versions change, and misalignment in either direction (too tight, throttling; too loose, wasted reserved capacity) has a real, ongoing cost either way. --- ## Common Mistakes and Interview Traps | Mistake | Why It's Wrong | Fix | |---|---|---| | Trying to `kill -9` a process stuck in `D` state | Uninterruptible sleep can't be interrupted by any signal, including `SIGKILL`, until the underlying I/O resolves | Investigate and fix the underlying I/O source (disk, NFS) instead of the process | | Assuming CFS is still "the" Linux scheduler on any current kernel | EEVDF has been the default for `SCHED_NORMAL` tasks since Linux 6.6 | Know EEVDF by name; the external tuning interface (nice, cgroups) is unchanged either way | | Evaluating a Kubernetes CPU limit's adequacy using average utilization alone | Cgroup quota enforcement operates per-period on instantaneous bursts, not a smoothed average — throttling can happen even with low average usage | Check `container_cpu_cfs_throttled_periods_total`/`cpu.stat` directly, not just the utilization-vs-limit graph | | Treating a high load average as always meaning "needs more CPU" | Linux load average also counts threads in uninterruptible (I/O) sleep, not just CPU contention | Cross-check load average against actual CPU utilization before concluding what kind of capacity problem it is | | Reaching for `SCHED_FIFO`/real-time priority to "make a thread faster" | Real-time scheduling classes override fairness entirely and can starve every normal thread if misused | Use `nice` for the common case of soft priority preference; reserve real-time classes for genuinely audited, time-critical code | | Setting CPU limits without understanding they cap burst capacity, not just average usage | A generous-looking limit relative to average usage can still throttle a bursty multi-threaded workload | Size limits (or decide to leave them unset) based on burst behavior, not average utilization alone | | Conflating `kube-scheduler` with the Linux kernel's CPU scheduler when triaging a "scheduling" complaint | They are two entirely separate systems solving different problems at different granularities | Clarify pod-placement vs. in-pod CPU-scheduling immediately before investigating either | | Assuming pinning a thread to a set of cores grants it exclusive access to them | Plain CPU affinity only narrows the eligible core set — fairness competition among everything else sharing those cores still applies | Confirm whether exclusive reservation (Kubernetes CPU Manager `static` policy) is actually configured, not just affinity | | Treating the aggregate context-switch rate as sufficient for capacity monitoring | Voluntary and involuntary switches can trend in opposite directions, canceling out in the total while still signaling real, worsening contention | Track voluntary and involuntary context-switch rates as separate metrics | | Deploying a Go, JVM, or similar service into a resource-limited container without checking runtime cgroup awareness | An older or unaware runtime can size its own thread pool from host CPU count, guaranteeing severe throttling under a tight cgroup limit | Confirm the runtime version/library respects the cgroup limit (Go 1.25+, `automaxprocs`, or a current JVM) before deploying into a small container | | Assuming a niced-down background job that's "making no progress" is a scheduling-priority problem | Nice values only affect competition for CPU among runnable threads — a thread stuck in `D`/`S` state isn't even competing for CPU at all | Check the process's actual state before adjusting priority further; a blocked thread needs its I/O path fixed, not a nicer nice value | | Assuming a wait-queue wakeup is instantaneous and free | Waking a thread still requires a full context switch (register restore, cold cache) before it resumes useful work | Treat a wakeup-heavy workload (many short blocking calls) as carrying real scheduling overhead, not a zero-cost operation | | Assuming scheduling domains guarantee perfectly even per-core utilization at all times | The kernel deliberately trades some short-term evenness for avoiding expensive cross-domain migrations | Cross-reference uneven per-core utilization against NUMA/socket topology before treating it as a bug | | Reading `cpu.shares`/`cpu.weight` values as if they were an absolute core count | Both are purely relative — meaningful only when multiple cgroups actively contend for the same cores | Check the absolute quota control (`cpu.max`/`cpu.cfs_quota_us`) separately for any hard-ceiling question | | Assuming a service's CPU limit is the only thing worth checking for a managed-runtime latency spike | GC cycles are real, independently-triggered scheduled CPU work layered on top of request-driven demand | Correlate spike timing against GC logs, not just request-volume or limit-vs-usage graphs | | Reaching for real-time (`SCHED_FIFO`/`SCHED_RR`) priority to fix a priority-inversion symptom instead of priority inheritance | Real-time priority changes who wins scheduling contention, but does nothing about a lock held by a lower-priority thread the high-priority thread is genuinely waiting on | Use a lock implementation supporting priority inheritance (`PTHREAD_PRIO_INHERIT`) for the specific lock involved | | Assuming an unexplained latency spike in a managed-runtime service is an application bug before checking GC logs | Garbage collection is real, scheduled CPU work that competes for the same quota as request-handling threads, and can produce spikes unrelated to any code path | Cross-reference spike timing against the runtime's GC logs before starting an application-level investigation | | Assuming cgroup v1 file names (`cpu.shares`, `cpu.cfs_quota_us`) are simply missing or the host is misconfigured when they're absent | The host may be running cgroup v2, which uses different file names (`cpu.weight`, `cpu.max`) for the same underlying concepts | Check `mount | grep cgroup2` to confirm which version is active before assuming a missing file means something is broken | --- ## Worked Practice Problems **Problem 1:** `checkout-service`'s p99 latency shows small, frequent stutters. The team's dashboard shows CPU utilization comfortably under the pod's CPU limit at all times. A teammate concludes the CPU limit isn't the cause and starts investigating the network layer instead. What would you check first, and why? **Answer:** Check the pod's cgroup throttling metric (`container_cpu_cfs_throttled_periods_total` or `cpu.stat`'s `nr_throttled`/`throttled_usec`) before ruling out CPU limits. Section 9 established that CFS/EEVDF quota enforcement operates on instantaneous, per-period bursts (typically 100ms windows), not on the smoothed average utilization a dashboard usually displays — a workload can be throttled thousands of times a day while its average utilization graph looks perfectly healthy. The symptom described (small, frequent stutters, not sustained high latency) is specifically the pattern CFS throttling produces, so it deserves direct verification before being ruled out based on a metric (average utilization) that structurally cannot reveal it. **Problem 2:** A background data-export job for `inventory-service` is niced down (`nice -n 15`) to avoid affecting foreground request handling, but the team notices it's making almost no progress even during periods when the host is nearly idle. Is the nice value the cause, and what would you check? **Answer:** A nice value alone is very unlikely to be the cause here — CFS/EEVDF fairness never fully starves a runnable thread; a niced-down process still receives CPU time proportional to its lower weight, and on a *nearly idle* host there's little contention for the nice value to even matter against. The more likely explanation is that the job is spending most of its time blocked (in `S` or `D` state) rather than being CPU-scheduled at all — worth checking its process state directly (`ps aux`) and, if it's in `D` state, investigating the I/O path it depends on (Part 4's territory) rather than adjusting scheduling priority further. Nice values control competition for the CPU specifically; they have no effect on a thread that isn't even runnable. **Problem 3:** A platform team is deciding whether to leave CPU limits unset on a bursty, latency-sensitive service, relying only on CPU requests for scheduling priority. What's the tradeoff, and what would make you recommend for versus against setting a limit? **Answer:** Leaving the limit unset means the service can use any spare capacity on the node when available (a real throughput benefit for a bursty workload), but loses the hard, predictable ceiling a limit provides — on a busy, multi-tenant node, an unset limit means this service's resource consumption is bounded only by its CPU *request*-driven share under contention, not by any hard cap, which can make capacity planning for its neighbors less predictable. The recommendation genuinely depends on the node's tenancy model: on a dedicated or lightly-shared node where "use whatever's free" is safe, leaving limits unset avoids Section 9's throttling risk entirely for a bursty workload. On a densely multi-tenant node where predictable per-service resource bounds matter more than any one service's peak throughput, a limit (sized generously enough to accommodate real burst behavior, not just average usage) is the safer choice. **Problem 4:** A `checkout-service` deploy introduces a shared in-memory rate-limiter guarded by a single mutex, used by both the ordinary request-handling threads (default priority) and a newly added, higher-priority fraud-detection thread. Under load, the fraud-detection thread occasionally shows latency spikes disproportionate to its high priority. What mechanism from this chapter is the most likely explanation, and what's the standard fix? **Answer:** This is a textbook priority inversion scenario (Section 15): the fraud-detection thread blocks waiting for the mutex whenever an ordinary-priority request-handling thread holds it, and if a third, medium-priority thread (a monitoring agent, a GC thread, anything scheduled between the two) keeps preempting the mutex-holding ordinary thread, the high-priority fraud-detection thread can be indirectly, unboundedly delayed by a thread it has no direct relationship with. The standard fix is priority inheritance — temporarily boosting the mutex-holder's priority to match the highest-priority waiter for as long as it holds the lock (`PTHREAD_PRIO_INHERIT` for POSIX mutexes), which prevents a medium-priority thread from being able to preempt the holder and indirectly stall the high-priority waiter. **Problem 5:** A team observes that a `catalog-service` batch-indexing pod, pinned via Kubernetes CPU Manager's `static` policy to 4 exclusive cores, still shows some CFS-throttling events in `cpu.stat`. A teammate argues this must be a monitoring bug, since exclusive pinning should mean no contention exists at all. Is the teammate right? **Answer:** No — exclusive core pinning (Section 21) removes those cores from the shared scheduling pool for *other* workloads, but it does nothing to change the pod's own cgroup CPU **limit** if one is still set. Even running completely alone on its 4 dedicated cores, the pod's threads can still exceed its own quota-per-period ceiling (Section 9) if they burst hard enough within a single period — pinning solves cross-workload contention, not a workload's own limit being set tighter than its genuine burst behavior requires. The fix, if the throttling is genuinely unwanted, is raising or removing the CPU limit itself (which is a separate cgroup control, `cpu.max`, from the exclusive-core reservation), not assuming pinning alone makes throttling impossible. **Problem 6:** A JVM-based service shows throttling that scales with request volume during normal traffic, but also shows occasional throttling spikes during low-traffic overnight periods when almost no requests are being handled at all. What's the most likely explanation for the overnight spikes specifically, and how would you confirm it? **Answer:** The most likely explanation is garbage collection running independently of request volume — a scheduled or threshold-triggered GC cycle (compaction, a full/major collection) can consume real CPU quota regardless of how much request traffic is actually flowing, since collection is driven by memory allocation and object lifetime, not directly by request count. Confirmation is straightforward: cross-reference the timestamps of the overnight throttling spikes against the JVM's own GC logs — an exact timing match confirms the collector as the cause. If confirmed, the fix isn't necessarily more CPU quota; it may be tuning the collector to run less disruptively (a different collection algorithm, adjusted heap sizing, or scheduling major collections for genuinely idle windows if the runtime supports it) rather than paying for headroom that's only needed during infrequent collection events. **Problem 7:** A host running cgroup v1 shows `cpu.shares` set to `2048` for one container and `1024` for its sibling. A teammate new to this host, more familiar with recently-provisioned cgroup v2 clusters, asks whether these values represent 2 and 1 CPU cores respectively. How would you correct that assumption? **Answer:** No — `cpu.shares` (and its cgroup v2 equivalent, `cpu.weight`) is a *relative* value, not an absolute core count; it only determines proportional CPU allocation when multiple cgroups are actively contending for the same cores. A `2048`-vs-`1024` split means the first container gets roughly twice the CPU time of the second *specifically under contention* — if the host is otherwise idle, both containers can use as much CPU as they need without either value ever coming into play at all. Absolute CPU ceilings are a completely separate control (`cpu.cfs_quota_us`/`cpu.cfs_period_us` under v1, `cpu.max` under v2) — conflating the relative-share mechanism with an absolute core count is a common, understandable mistake precisely because the numbers (2048, 1024) superficially resemble a core count when they aren't one. --- ## Summary and What's Next **Quick reference — key terms from this chapter:** | Term | One-line definition | |---|---| | Run queue | The per-core set of threads currently ready and waiting to be scheduled | | CFS / EEVDF | The default Linux scheduling algorithms for ordinary tasks — fairness via vruntime (CFS) or virtual deadlines (EEVDF, since kernel 6.6) | | Nice value | A per-process priority weight (-20 to +19) controlling its relative share of CPU time under CFS/EEVDF | | cgroup CPU quota | A hard, per-period CPU time ceiling enforced on a group of processes — the mechanism behind every Kubernetes CPU limit | | CPU throttling | A process being paused once its cgroup quota is exhausted within the current period, even if longer-term average usage looks low | | Load average | Linux's count of runnable *and* uninterruptible-sleep threads — not a pure CPU-utilization metric | | Preemption | A running thread being interrupted so a higher-priority or longer-waiting thread can run | | Priority inversion | A high-priority thread indirectly blocked by a medium-priority one, via a lock held by a low-priority thread that can't finish | | cpu.weight | The cgroup v2 relative CPU share value (1-10000) derived from a Kubernetes CPU request | | Scheduling domain | The kernel's hierarchy of core groupings (SMT, then socket, then NUMA) used to bias load balancing toward cheaper migrations | | CONFIG_HZ / tickless | The rate of the kernel's scheduling timer interrupt, and the mechanism (NO_HZ) that suppresses it on idle or single-thread cores | | cgroup v1 / v2 | Two generations of the kernel's control-group interface for grouped resource limits — different file names and hierarchy model, same underlying scheduling enforcement | | Stop-the-world GC | A garbage-collection pause that halts every application thread, as opposed to a concurrent collector running alongside them on its own threads | | Wait queue | A kernel structure holding threads blocked on a specific event, entirely off any run queue until woken | | GOMAXPROCS | The Go runtime's OS-thread parallelism setting — historically host-CPU-aware only, container/cgroup-aware by default since Go 1.25 | | kube-scheduler | The Kubernetes control-plane component deciding pod-to-node placement — distinct from, and unaware of, the in-node Linux kernel scheduler | This chapter's mechanisms don't operate in isolation — a niced-down batch job, a throttled container, and a GC-paused thread can all be happening on the same host at the same moment, and distinguishing which one actually explains a given latency spike is the practical skill this chapter has built toward, one mechanism at a time. The scheduler decides which thread runs, on which core, for how long — but it can only schedule threads that actually exist and have memory to execute in. Every mechanism in this chapter (vruntime, nice values, cgroup quotas) assumes memory access is already resolved and instantaneous, which is a simplification Part 3 removes entirely: physical versus virtual memory, page tables, swap, and the page cache — the layer that determines whether a scheduled thread's next memory access is actually fast, or a hidden, scheduler-invisible stall.