Compute Engine & Autoscaling

Assumes you're comfortable with Part 3's IAM model (service accounts, roles, bindings) — this chapter is the first place a workload's identity actually attaches to a running machine.

Table of Contents#

1. What This Chapter Covers
2. Choosing a Machine Family: E2, N4, and C4
3. Custom Machine Types
4. Launching an Instance: Images, Startup Scripts, and Availability Policy
5. Persistent Disk and Hyperdisk — Choosing Storage for Compute Engine
6. OS Login — Centralized, IAM-Governed SSH Access
7. VM Manager — Patch, Config, and OS Inventory Management
8. Spot VMs — Deep Discounts With a Real Trade-off
9. Managed Instance Groups and Instance Templates
10. Autoscaling Policies: Metrics, Predictive, and Scheduled
11. Health Checks — What "Unhealthy" Actually Means to a MIG
12. Worked Cost Comparison: Standard vs. Spot for the Overflow Group
13. Snapshots and Images — Backup and Golden-Image Strategy
14. GPUs and TPUs — When and How to Attach Them
15. A Full Worked Example: Meridian's GPS-Ingestion Fleet End to End
16. Compute Terminology Map
17. Real-World Scenario: The Autoscaling Policy That Fought Itself
18. Regional vs. Zonal MIGs, and Stateful Workloads
19. Configuration Management With VM Manager's OS Config
20. Second Real-World Scenario: The Golden Image That Drifted From Its Source
21. Third Real-World Scenario: A Zone Outage During Peak Hours
22. Pre-Flight Checklist: Is This Compute Design Production-Ready?
23. Chapter Recap: How the Pieces Connect
24. Common Mistakes and Interview Traps
25. Worked Practice Problems
26. Summary and What's Next

What This Chapter Covers#

Compute Engine is GCP's raw virtual machine service — the layer with the fewest guardrails and the most manual responsibility, sitting at the far end of Part 1's shared-responsibility spectrum from Cloud Run. This chapter covers everything the ACE exam's compute objectives test: choosing the right machine type and disk, launching and managing instances, OS Login and VM Manager for day-2 operations, Spot VMs for cost, and managed instance groups with autoscaling — the mechanism that turns a single VM definition into a fleet that grows and shrinks with real load.

🎯 By the end of this chapter, you'll be able to size a Compute Engine workload correctly on the first attempt, choose between Spot and standard VMs with real numbers behind the decision, and configure autoscaling that reacts to actual demand instead of a guessed fixed fleet size.

Meridian's GPS-ingestion fleet — introduced in Part 1's architecture diagram — is this chapter's primary running example: a Compute Engine managed instance group that consumes GPS pings from Pub/Sub, the one Meridian workload deliberately kept on raw VMs rather than a serverless platform, because the previous chapters' whole point was building the operational muscle this chapter now puts to use.

A theme worth naming upfront, because it shapes nearly every decision in this chapter: raw Compute Engine trades Google's operational involvement for your own control, the leftmost end of Part 1's shared-responsibility spectrum. Every mechanism below — OS Login instead of manual SSH keys, VM Manager instead of manual patch tracking, managed instance groups instead of hand-maintained individual VMs — exists specifically to claw back some of the operational convenience a fully-managed platform gives you by default, without giving up the control that made you choose raw Compute Engine in the first place. Part 5's serverless options make the opposite trade explicitly; keeping that contrast in mind while reading this chapter makes the "why does GCP make me configure this manually" moments land as a deliberate trade-off rather than an omission.

Choosing a Machine Family: E2, N4, and C4#

GCP's current machine families split cleanly by what they optimize for, and "just pick something and see" is a real cost mistake, not a harmless placeholder.

E2 is the right default for anything cost-sensitive and not latency-critical; C4 earns its price premium only when single-thread performance or very high vCPU counts genuinely matter to the workload.

# Launch an N4 instance — Meridian's current default for the
# GPS-ingestion fleet
gcloud compute instances create gps-worker-1 \
  --machine-type=n4-standard-8 \
  --zone=us-central1-a \
  --image-family=debian-12 \
  --image-project=debian-cloud

From the Trenches: The C4 Instance Nobody Needed#

An engineer evaluating Compute Engine for the first time defaulted to a c4-standard-16 for a low-traffic internal admin dashboard, reasoning "C4 is the newest, it must be the best default." The dashboard served a handful of internal users and never came close to needing C4's single-thread performance ceiling — an e2-medium would have handled the actual load with room to spare, at roughly a quarter of the cost. The immediate cause was picking a machine family by recency bias rather than a workload's actual profile; the deeper lesson, consistent with this chapter's own quadrant chart, is that "newest" and "best default" are not the same claim — a machine family is a fit-for-workload decision, not a version number to maximize.

Beyond General Purpose: Compute- and Memory-Optimized Families#

The three families compared above (E2, N4, C4) cover general-purpose workloads; two further families exist for genuinely specialized profiles worth knowing even though Meridian doesn't use either yet:

Reaching for either without a workload that actually needs the specialized ratio repeats the exact C4-by-recency-bias mistake above, just one level more specialized — the decision discipline (measure first, size second) is identical regardless of which family is under consideration.

Custom Machine Types#

When a predefined machine type's vCPU-to-memory ratio doesn't match a workload's actual profile, a custom machine type lets you specify exact vCPU and memory counts instead of rounding up to the next predefined size.

# A memory-heavy workload that doesn't need 8 full vCPUs' worth of
# compute — a custom type avoids paying for CPU capacity the
# workload will never use
gcloud compute instances create reporting-batch \
  --custom-cpu=4 --custom-memory=32GB \
  --zone=us-central1-a \
  --image-family=debian-12 --image-project=debian-cloud

Launching an Instance: Images, Startup Scripts, and Availability Policy#

# A more complete launch — a startup script for bootstrapping,
# the correct service account (never the default Compute Engine
# service account for anything beyond a quick test), and explicit
# availability policy
gcloud compute instances create gps-worker-1 \
  --machine-type=n4-standard-8 \
  --zone=us-central1-a \
  --image-family=debian-12 --image-project=debian-cloud \
  --service-account=gps-worker@meridian-shipment-prod.iam.gserviceaccount.com \
  --scopes=cloud-platform \
  --no-address \
  --metadata-from-file=startup-script=bootstrap-gps-worker.sh \
  --maintenance-policy=MIGRATE

Three flags above are worth calling out individually because each maps directly to an earlier chapter's principle: --service-account uses the narrowly-scoped gps-worker identity from Part 3, never the broad project-default Compute Engine service account; --no-address skips a public IP entirely, enforcing Part 1's org policy at the instance level rather than depending on the policy alone to catch a mistake; --maintenance-policy=MIGRATE lets GCP live-migrate the VM to different underlying hardware during host maintenance rather than terminating it, the right default for anything not explicitly designed to tolerate termination (Spot VMs, covered later, are the deliberate exception).

Persistent Disk and Hyperdisk — Choosing Storage for Compute Engine#

Compute Engine's block storage comes in two generations — Persistent Disk (the long-standing default) and Hyperdisk (the newer, higher-performance option) — and picking between them is a real, workload-specific decision, not just "use whichever is newer."

# Attach a Hyperdisk Balanced volume, sized independently from its
# performance tier — a real structural difference from classic
# Persistent Disk, where IOPS scales with size alone
gcloud compute disks create gps-worker-data \
  --type=hyperdisk-balanced \
  --size=200GB \
  --provisioned-iops=10000 \
  --provisioned-throughput=200 \
  --zone=us-central1-a

OS Login — Centralized, IAM-Governed SSH Access#

OS Login replaces per-instance SSH key management with IAM-governed access, tying who can SSH into a VM directly to the IAM roles covered in Part 3 rather than a separately-managed key file.

# Enable OS Login at the project level — the org policy from Part 1
# (constraints/compute.requireOsLogin) can enforce this org-wide
gcloud compute project-info add-metadata \
  --metadata=enable-oslogin=TRUE

# Grant a human SSH access via IAM, not a manually-distributed key
gcloud projects add-iam-policy-binding meridian-shipment-prod \
  --member="group:platform-all-admin@meridianlogistics.com" \
  --role="roles/compute.osLogin"

# Grant SUDO-capable SSH access specifically — a narrower default
# would use roles/compute.osLogin above without this broader role
gcloud projects add-iam-policy-binding meridian-shipment-prod \
  --member="group:platform-all-admin@meridianlogistics.com" \
  --role="roles/compute.osAdminLogin"

Without OS Login, SSH access depends on manually placing a public key into each instance's (or the project's) metadata — a real management burden at fleet scale, and one with no connection to Part 3's IAM system at all: revoking a departed engineer's IAM roles does nothing to their previously-distributed SSH key unless someone separately remembers to remove it from every instance's metadata. OS Login closes exactly this gap — revoking roles/compute.osLogin from a departed engineer's group membership immediately removes their SSH access everywhere, the same "one control point" benefit Part 3 built its entire IAM design around.

VM Manager — Patch, Config, and OS Inventory Management#

VM Manager is a suite of three related tools — patch management, OS configuration management, and OS inventory — that turn "did we patch every VM for this CVE" from a manual fleet-wide check into a queryable, automatable answer.

# Enable VM Manager for a project
gcloud compute instances ops-agents policies create patch-policy \
  --project=meridian-shipment-prod

# Deploy a patch job across a fleet, on demand
gcloud compute os-config patch-jobs execute \
  --project=meridian-shipment-prod \
  --instance-filter-groups="gps-ingestion" \
  --description="Emergency patch for CVE-2026-XXXX"

# Query current OS inventory across the fleet — genuinely useful
# during exactly the kind of CVE-response scramble the patch job above addresses
gcloud compute os-config inventories list \
  --project=meridian-shipment-prod --location=us-central1-a

Spot VMs — Deep Discounts With a Real Trade-off#

A Spot VM runs on GCP's spare compute capacity at a steep discount (typically 60-91% off standard pricing) in exchange for accepting that GCP can reclaim it with only 30 seconds' notice, whenever that capacity is needed elsewhere.

# Launch a Spot VM — note the explicit MIGRATE policy is NOT
# available here; Spot instances are always TERMINATE on preemption
gcloud compute instances create batch-worker-spot \
  --provisioning-model=SPOT \
  --instance-termination-action=STOP \
  --machine-type=n4-standard-4 \
  --zone=us-central1-a \
  --image-family=debian-12 --image-project=debian-cloud

Realistic Scenario: The Spot Preemption During a Load Spike#

During a regional weather event that spiked delivery-tracking traffic well above normal, Meridian's autoscaler added Spot capacity to handle the surge — and a batch of those Spot instances were preempted roughly forty minutes later when GCP's own capacity demand shifted, right in the middle of the traffic spike. Because the ingestion workers only ever hold an in-flight message for the duration of processing it (never accumulating unacknowledged state across a longer window), the preempted instances' in-flight messages simply redelivered to the remaining fleet within Pub/Sub's normal acknowledgment-deadline window — a brief redelivery blip, not a data loss event, and invisible to end users tracking their shipments. This is the concrete payoff of the "design for preemption from the start" principle above: the cost savings were real, and the failure mode Spot VMs guarantee will eventually happen cost Meridian nothing, because the workload's own architecture had already assumed it would.

Managed Instance Groups and Instance Templates#

A Managed Instance Group (MIG) maintains a fleet of identical VM instances from one instance template, automatically replacing unhealthy instances and providing the foundation autoscaling builds on.

# Define the instance template once — every instance in the MIG is
# created from this exact configuration
gcloud compute instance-templates create gps-worker-template \
  --machine-type=n4-standard-8 \
  --image-family=debian-12 --image-project=debian-cloud \
  --service-account=gps-worker@meridian-shipment-prod.iam.gserviceaccount.com \
  --scopes=cloud-platform \
  --no-address \
  --metadata-from-file=startup-script=bootstrap-gps-worker.sh

# Create the MIG from the template, across multiple zones for
# resilience against a single zone failure
gcloud compute instance-groups managed create gps-worker-mig \
  --template=gps-worker-template \
  --size=24 \
  --region=us-central1

# Configure a health check-driven rolling update policy — how the
# MIG replaces instances without a thundering-herd all-at-once restart
gcloud compute instance-groups managed update gps-worker-mig \
  --region=us-central1 \
  --health-check=gps-worker-health-check \
  --initial-delay=120

Every instance is created identically from one template and spread across zones — a MIG never "hand-tunes" one instance differently from its siblings; changing configuration means updating the template and rolling it out.

What Actually Happens Inside the Startup Script#

The bootstrap-gps-worker.sh referenced throughout this chapter's commands has been treated as a black box so far — worth showing the real content once, since a startup script is where most of a MIG's actual application-specific setup lives, everything the instance template's own flags don't cover:

#!/bin/bash
# bootstrap-gps-worker.sh — runs once, automatically, on every
# instance's first boot, whether created manually, by a MIG, or
# recreated after a health-check failure
set -euo pipefail

# Fetch the application's own config from a source outside the
# image itself — never bake environment-specific values into the
# golden image, since the same image serves multiple environments
CONFIG_BUCKET="gs://meridian-shared-config/gps-worker/prod.env"
gsutil cp "${CONFIG_BUCKET}" /etc/gps-worker/prod.env

# Install and start the Ops Agent (Part 8 covers this fully) —
# every instance reports metrics/logs from the moment it boots,
# never as a manual follow-up step someone might forget
curl -sSO https://dl.google.com/cloudagents/add-google-cloud-ops-agent-repo.sh
bash add-google-cloud-ops-agent-repo.sh --also-install

# Start the actual application as a systemd service, reading the
# config fetched above — the application itself is baked into the
# golden image; only environment-specific config is fetched at boot
systemctl enable gps-worker.service
systemctl start gps-worker.service

A shutdown script runs on the reverse path — when the MIG or a manual action terminates the instance — and matters specifically for graceful draining:

#!/bin/bash
# shutdown-gps-worker.sh — has a limited window (default 90 seconds
# for a standard VM, non-negotiable at 30 seconds for Spot) to
# finish in-flight work before the instance is forcibly terminated
set -euo pipefail

# Tell the load balancer / MIG this instance is draining, so new
# work stops routing here while existing in-flight messages finish
systemctl stop gps-worker.service --no-block
sleep 20   # grace period for in-flight Pub/Sub message acks to complete
systemctl kill gps-worker.service

Autoscaling Policies: Metrics, Predictive, and Scheduled#

An autoscaling policy can combine multiple signals — CPU utilization, a custom Cloud Monitoring metric, load-balancer capacity, and a time-based schedule — and the autoscaler always sizes the group to the largest recommendation any single signal produces.

# A policy combining CPU utilization with the business-hours
# scaling schedule introduced in Part 2's cost-review worked example
gcloud compute instance-groups managed set-autoscaling gps-worker-mig \
  --region=us-central1 \
  --min-num-replicas=8 \
  --max-num-replicas=48 \
  --target-cpu-utilization=0.6 \
  --scale-in-control-max-scaled-in-replicas-percent=10

When multiple signals disagree, the autoscaler always takes the larger recommendation — a design that biases toward availability over cost when signals conflict.

--scale-in-control deliberately caps how fast the group can shrink in a single autoscaling evaluation — without it, a brief metric dip can trigger an aggressive scale-in immediately followed by a scale-back-up once the dip passes, a wasteful oscillation pattern sometimes called "flapping." Capping scale-in to 10% of the fleet per evaluation smooths this out at a small cost in scale-down responsiveness.

Cooldown Periods and Avoiding Premature Scale-In#

Every newly-created instance needs time to actually start serving before its metrics are meaningful — scaling it back in seconds after creation because it hasn't yet ramped up to normal utilization would be measuring an instance that hasn't finished starting, not one that's genuinely underloaded.

# initial-delay tells the autoscaler to ignore a new instance's
# metrics for this many seconds after creation — long enough for
# gps-worker's own startup script and warm-up period to complete
gcloud compute instance-groups managed update gps-worker-mig \
  --region=us-central1 \
  --initial-delay=180

Set this to genuinely reflect how long the workload takes to reach steady-state performance after boot — Meridian's gps-worker needs roughly two minutes between systemd starting the service and it reaching normal throughput (config fetch, connection warm-up to Pub/Sub), so 180 seconds gives real margin rather than the bare minimum. An initial-delay set too short repeats this chapter's own flapping lesson from a different angle: the autoscaler judges a still-warming-up instance by metrics that don't yet reflect its real capacity, potentially triggering an unnecessary scale-in of an instance that was about to become useful.

Predictive autoscaling goes further, using historical load patterns to scale ahead of an anticipated spike rather than reacting after utilization has already climbed — genuinely valuable for a workload with a predictable daily/weekly pattern, which describes Meridian's business-hours traffic well:

gcloud compute instance-groups managed set-autoscaling gps-worker-mig \
  --region=us-central1 \
  --min-num-replicas=8 --max-num-replicas=48 \
  --target-cpu-utilization=0.6 \
  --mode=ON \
  --cpu-utilization-predictive-method=OPTIMIZE_AVAILABILITY

Health Checks — What "Unhealthy" Actually Means to a MIG#

Every reference to a MIG "automatically recreating an unhealthy instance" earlier in this chapter depends on a health check actually being configured correctly — an under-specified health check is one of the most common reasons a MIG behaves unpredictably in production.

# An HTTP health check with real, deliberately-chosen timing —
# not the tightest possible values, which risks exactly the
# patch-job-triggers-cascade failure mode covered earlier
gcloud compute health-checks create http gps-worker-health-check \
  --port=8080 \
  --request-path=/healthz \
  --check-interval=30s \
  --timeout=10s \
  --healthy-threshold=2 \
  --unhealthy-threshold=3

The four timing parameters directly trade off detection speed against false-positive risk: a short check-interval and low unhealthy-threshold detects a genuine failure fast, but also more readily misinterprets a brief, harmless blip (a GC pause, a momentary CPU spike during a patch reboot) as a real failure, triggering an unnecessary — and, per this chapter's earlier warning, potentially cascading — instance recreation.

Worked Cost Comparison: Standard vs. Spot for the Overflow Group#

Putting real numbers behind the Spot-VM discount claim earlier in this chapter, using Meridian's actual n4-standard-8 overflow group at a representative regional on-demand rate (illustrative pricing — always confirm current rates via the official pricing calculator before a real budget decision):

The overflow-on-Spot design in this chapter's worked example isn't just operationally sound — it's roughly 58% cheaper than running the full peak-sized fleet on standard VMs continuously, while still meeting peak demand when it actually arrives. This is the concrete number behind Part 2's abstract cost-optimization framing: a design choice made for architectural reasons (Pub/Sub's redelivery guarantee making Spot safe) also happens to be the financially correct one, which is the ideal outcome rather than a coincidence worth being suspicious of — a workload's actual tolerance for interruption and its cost profile are often aligned, since both trace back to the same underlying question of how much guaranteed capacity a workload genuinely needs at every moment versus only at genuine peaks.

Snapshots and Images — Backup and Golden-Image Strategy#

# A one-off snapshot — the disk-level backup mechanism
gcloud compute disks snapshot gps-worker-data \
  --snapshot-names=gps-worker-data-backup-2026-09 \
  --zone=us-central1-a

# A recurring snapshot schedule, attached to a disk — automates
# the backup cadence instead of depending on someone remembering
gcloud compute resource-policies create snapshot-schedule daily-backup-policy \
  --region=us-central1 \
  --max-retention-days=30 \
  --daily-schedule --start-time=03:00
gcloud compute disks add-resource-policies gps-worker-data \
  --resource-policies=daily-backup-policy --zone=us-central1-a

# A custom image built FROM a snapshot — the golden-image pattern,
# baking a known-good configuration in rather than re-running a
# startup script from scratch on every boot
gcloud compute images create gps-worker-golden-v3 \
  --source-disk=gps-worker-data --source-disk-zone=us-central1-a \
  --family=gps-worker-golden

GPUs and TPUs — When and How to Attach Them#

GPUs (general-purpose parallel compute, common for ML training/inference and some rendering workloads) and TPUs (Google's own tensor-processing hardware, purpose-built for ML) attach to specific machine types, not universally, and both carry real availability and quota considerations worth planning around before a launch, not during one.

# Attach a GPU to a compatible instance
gcloud compute instances create ml-inference-1 \
  --machine-type=n1-standard-8 \
  --accelerator=type=nvidia-tesla-t4,count=1 \
  --maintenance-policy=TERMINATE \
  --zone=us-central1-a \
  --image-family=debian-12 --image-project=debian-cloud

TPUs are Google's own custom silicon, purpose-built for the matrix-multiplication-heavy workloads at the core of neural network training and inference, and reached differently from GPUs — as their own dedicated resource type rather than an accelerator attached to a general-purpose VM:

# TPUs are provisioned as their own resource, not attached to an
# arbitrary Compute Engine instance the way GPUs are
gcloud compute tpus tpu-vm create route-optimizer-tpu \
  --zone=us-central1-a \
  --accelerator-type=v5litepod-8 \
  --version=tpu-vm-v5lite-pod

Meridian has no GPU/TPU workload today — the decision table above is worth having ready regardless, since a future route-optimization ML model is a realistic next step for the analytics side of the business this course's throughline company represents, and the CPU-only row is worth taking seriously rather than assuming any ML workload automatically needs an accelerator.

A Full Worked Example: Meridian's GPS-Ingestion Fleet End to End#

Putting every mechanism in this chapter together — Meridian's actual current compute configuration for the GPS-ingestion pipeline:

# 1. A scoped service account (Part 3) — already created, referenced here
# 2. An instance template on N4, no public IP, OS Login enforced via org policy
gcloud compute instance-templates create gps-worker-template-v4 \
  --machine-type=n4-standard-8 \
  --image-family=debian-12 --image-project=debian-cloud \
  --service-account=gps-worker@meridian-shipment-prod.iam.gserviceaccount.com \
  --scopes=cloud-platform --no-address \
  --metadata-from-file=startup-script=bootstrap-gps-worker.sh

# 3. A standard-VM MIG for the reliable floor, min 8 instances
gcloud compute instance-groups managed create gps-worker-mig-standard \
  --template=gps-worker-template-v4 --size=8 --region=us-central1

# 4. A SEPARATE Spot-VM MIG for elastic overflow capacity — kept
# distinct from the standard MIG specifically so a Spot preemption
# event never touches the reliable floor
gcloud compute instance-groups managed create gps-worker-mig-spot \
  --template=gps-worker-template-spot --size=0 --region=us-central1

# 5. Metric + schedule-based autoscaling on the standard floor
gcloud compute instance-groups managed set-autoscaling gps-worker-mig-standard \
  --region=us-central1 --min-num-replicas=8 --max-num-replicas=24 \
  --target-cpu-utilization=0.6

# 6. Aggressive metric-based autoscaling on the Spot overflow group —
# scales from zero, absorbs the peaks the standard floor doesn't cover
gcloud compute instance-groups managed set-autoscaling gps-worker-mig-spot \
  --region=us-central1 --min-num-replicas=0 --max-num-replicas=24 \
  --target-cpu-utilization=0.5

# 7. Daily snapshot policy on any stateful disk, golden image
# refreshed monthly, VM Manager patch policy applied fleet-wide

This design directly answers the earlier decision table's "partially" for Spot fit: the standard MIG guarantees a reliable floor that never depends on Spot availability, while the Spot MIG absorbs everything above that floor at a steep discount, accepting occasional preemption precisely because Pub/Sub's redelivery guarantee makes that acceptable for this specific workload.

Compute Terminology Map#

Where this mapping holds up well: the underlying concepts (a scaling group, a discounted reclaimable tier, a reusable image) are genuinely similar shapes across all three clouds, more so than the resource-hierarchy or IAM concepts earlier chapters compared — compute fleet management has converged more across providers than identity models have.

Real-World Scenario: The Autoscaling Policy That Fought Itself#

Three months after the standard/Spot MIG split from this chapter's worked example went live, Ana noticed the Spot overflow group oscillating between roughly 4 and 18 instances every few minutes during a period of genuinely steady traffic — no real load change was happening, but the fleet size kept swinging. The autoscaling policy's target-cpu-utilization was set correctly, but the underlying metric it read (average CPU across the group) was itself noisy at low instance counts: with only 4-6 instances active, a single instance's normal CPU variance moved the group average by enough to repeatedly cross the scaling threshold in both directions, triggering the policy to add instances, then immediately remove them again once the average settled, then repeat.

The fix combined two changes from this chapter, applied together rather than either alone: raising --min-num-replicas on the Spot group from 0 to 4 (so the average was never computed across a tiny, noise-prone sample) and applying --scale-in-control-max-scaled-in-replicas-percent=15 to cap how fast the group could shrink in one evaluation cycle. Neither change alone fully resolved it — the minimum-replica floor reduced the noise, but the scale-in cap was what actually stopped the oscillation from being visible as repeated flapping once the noise was reduced but not eliminated.

Regional vs. Zonal MIGs, and Stateful Workloads#

A MIG can be zonal (all instances in one zone) or regional (spread automatically across multiple zones within a region) — the choice directly trades off blast radius against a small amount of added complexity, and Meridian's earlier examples in this chapter already default to the regional form for exactly this reason.

# The regional form used throughout this chapter — GCP decides the
# per-zone split automatically based on capacity and even distribution
gcloud compute instance-groups managed create gps-worker-mig \
  --template=gps-worker-template --size=24 --region=us-central1

Most MIGs, including every one in this chapter, are stateless — any instance can be destroyed and recreated identically from the template with no data loss, because the workload's actual state (Pub/Sub's own message queue, in Meridian's case) lives outside the instance entirely. A stateful MIG exists for the narrower case where each individual instance in the group needs to preserve its own identity or disk across recreation — a per-instance disk that survives instance deletion, or a fixed hostname/IP an instance keeps even after being recreated:

# A stateful MIG configuration — each instance keeps its own named
# disk across recreation, instead of a template-fresh disk every time
gcloud compute instance-groups managed create legacy-stateful-mig \
  --template=legacy-template --size=3 --zone=us-central1-a \
  --stateful-disk=device-name=data-disk,auto-delete=never

Configuration Management With VM Manager's OS Config#

Beyond the patch-management functionality covered earlier, VM Manager's OS Config Management applies a declarative configuration policy across a fleet — closer in spirit to Part 2's Config Connector than to a one-off patch job, and genuinely useful for enforcing a baseline that shouldn't depend on what a golden image happened to contain at build time.

# os-policy.yaml — enforce that a specific monitoring agent package
# is always installed and a specific config file always matches this
# content, regardless of what the base image shipped with
osPolicies:
  - id: enforce-monitoring-agent
    mode: ENFORCEMENT
    resourceGroups:
      - resources:
          - id: install-agent
            pkg:
              desiredState: INSTALLED
              apt:
                name: google-cloud-ops-agent
          - id: agent-config
            file:
              path: /etc/google-cloud-ops-agent/config.yaml
              state: PRESENT
              content: |
                logging:
                  receivers:
                    gps_worker_logs:
                      type: files
                      include_paths: ["/var/log/gps-worker/*.log"]

gcloud compute os-config os-policy-assignments create enforce-monitoring \
  --project=meridian-shipment-prod \
  --location=us-central1-a \
  --os-policy-assignment-file=os-policy.yaml \
  --instance-filter-inclusion-labels=team=platform

Second Real-World Scenario: The Golden Image That Drifted From Its Source#

Six months after Meridian's golden-image pipeline (this chapter's snapshot/image section) was set up, a security review found that three GPS-worker instances were running a version of a monitoring agent two minor versions behind what the current golden image shipped — despite all three having booted from what everyone assumed was the current image. The actual cause: the instances had been running long enough (through several autoscaling cycles that never happened to recreate them, since MIG recreation only happens on health-check failure or a deliberate rolling update) that they predated the last two golden-image refreshes entirely, and nothing had ever forced them to actually pick up the newer image.

This is the precise gap OS Config Management's ENFORCEMENT mode closes that a golden image alone cannot: a golden image only affects instances created after it's built, while a long-lived instance that happens to survive multiple image refreshes drifts further behind with each one, invisibly, until something like a security review checks actual running versions against the intended baseline. Meridian's fix combined both mechanisms rather than replacing one with the other — the golden image stays the fast path for new instance creation, while an OS Config policy continuously enforces the current agent version on already-running instances regardless of how long they've been up, closing the gap between "correct at boot" and "correct right now."

Third Real-World Scenario: A Zone Outage During Peak Hours#

Meridian's regional MIG design faced its first real test when us-central1-a — one of the three zones the GPS-worker fleet spreads across — experienced a partial outage during a Tuesday afternoon peak period. Roughly a third of the fleet's running instances (the ones GCP had placed in the affected zone) became unreachable within the same few minutes.

The regional MIG doesn't need to be told to avoid the bad zone — GCP's own placement logic for new instances naturally routes around a zone reporting failures, the entire reason Part 1's "choose a region, not a single zone" framing mattered.

Because the fleet was already regional (this chapter's own default recommendation) rather than pinned to one zone, the MIG's health checks caught the unhealthy instances within their configured detection window and began recreating replacements automatically — GCP's own instance-placement logic naturally avoided creating new instances in the still-degraded zone, without any manual intervention from Meridian's team. The remaining two-thirds of the fleet, spread across the two unaffected zones, absorbed the traffic that would otherwise have gone to the lost third while replacements spun up.

The one manual action the on-call engineer took was confirming — not fixing — that the autoscaler's aggregate CPU-utilization metric had correctly triggered additional scale-out on the two healthy zones to compensate for the temporarily reduced capacity, which it had, automatically, without needing the zone outage to be specifically detected as its own event type. This is the actual payoff of the "regional over zonal" and "combine a metric signal with a scheduled floor" recommendations made earlier in this chapter — neither recommendation was made with a zone outage specifically in mind, yet both mechanisms, designed for entirely different reasons (blast-radius reduction and cost efficiency, respectively), turned out to be exactly what made this failure a non-event instead of an incident requiring a war room.

Pre-Flight Checklist: Is This Compute Design Production-Ready?#

• Machine family chosen based on a measured workload profile, not recency bias or a guess
• No instance launched with the default Compute Engine service account
• OS Login enforced, with no per-instance SSH keys distributed manually
• Every stateful disk covered by a scheduled snapshot policy
• Golden images refreshed on a real cadence, not built once and left stale
• Spot VMs used only for workloads explicitly designed to tolerate 30-second-notice preemption
• A reliable standard-VM floor exists separately from any Spot overflow capacity
• Autoscaling combines a real metric signal with a scheduled floor, not either alone
• VM Manager patch policy applied fleet-wide, tested against staging before production
• Health checks target a real capability-to-serve endpoint, not a trivial always-passing one
• MIGs are regional, not zonal, for any workload that needs to survive a single-zone outage
• Startup and shutdown scripts are tested against both standard and Spot timing windows if the same template serves both
• initial-delay reflects the workload's actual warm-up time, not the autoscaler's default

A quick reference for which section of this chapter to revisit for a specific real-world symptom:

Chapter Recap: How the Pieces Connect#

Common Mistakes and Interview Traps#

Worked Practice Problems#

1. Meridian's GPS-ingestion fleet currently runs entirely on standard N4 VMs. Devon proposes moving the entire fleet to Spot VMs to cut compute costs by roughly 70%. What's the correct response, and what design would actually capture most of that savings safely?

Moving the entire fleet to Spot is the wrong move — it removes any guaranteed capacity floor, meaning a widespread preemption event (GCP reclaiming capacity broadly, not just a few instances) could leave the ingestion pipeline severely under-provisioned with no reliable fallback. The safer design, matching this chapter's worked example, splits into two separate MIGs: a standard-VM floor sized to the workload's reliable baseline need, and a Spot-VM group handling elastic overflow above that floor. This captures most of the discount (the overflow capacity, which is often the larger portion during peak periods) while keeping a guaranteed floor that never depends on Spot availability.

2. An autoscaling policy combines a CPU-utilization target with a custom queue-depth metric. During a specific evaluation, CPU utilization suggests scaling to 20 instances while queue depth suggests scaling to 35. How many instances does the autoscaler actually provision, and why does GCP's autoscaler default to this behavior rather than averaging the two signals?

35 — the autoscaler always takes the largest recommendation across every configured signal, never an average. This default biases toward availability over cost when signals disagree: averaging could under-provision relative to whichever single signal is correctly detecting real strain (here, queue depth showing a backlog CPU utilization alone doesn't yet reflect), risking a user-facing slowdown to save a comparatively small amount of compute cost.

3. A VM Manager patch job is scheduled to run against the production GPS-ingestion MIG tonight. What should be confirmed about the MIG's configuration before running it, and why does this matter specifically for a MIG rather than a set of standalone VMs?

Confirm the MIG's health check and rolling-update policy are correctly configured before running the patch job — specifically, that the health check has a reasonable grace period that won't misinterpret an in-progress reboot as a failed instance. This matters specifically for a MIG (versus standalone VMs) because a MIG automatically recreates any instance its health check considers unhealthy — if the health check is too aggressive relative to how long a patch-triggered reboot actually takes, the patch job's own reboots could trigger a cascade of unnecessary instance recreations, turning a routine patch into a fleet-wide disruption the same tool was meant to apply safely.

4. A security review finds several GPS-worker instances running an outdated monitoring agent version, despite the golden image having been refreshed twice since those instances first booted. Why didn't the image refresh reach them, and what mechanism from this chapter closes the gap without requiring every instance to be manually recreated?

A golden image only affects instances created after the image is refreshed — it has no effect on instances that are already running and simply never happen to get recreated (a MIG only recreates an instance on health-check failure or a deliberate rolling update, neither of which necessarily happens on the same cadence as an image refresh). The gap is closed by an OS Config Management policy in ENFORCEMENT mode, which continuously corrects configuration and package-version drift on already-running instances regardless of how long they've been up — the golden image stays the fast path for new instance creation, while OS Config handles instances that predate the latest refresh, without needing a disruptive fleet-wide forced recreation just to pick up a monitoring agent update.

5. Meridian is evaluating a new ML-based route-optimization model. Before deciding between a GPU-attached instance and a TPU, what question should be answered first, and why might the answer be "neither"?

Whether the model's actual inference workload is heavy enough to need hardware acceleration at all — a surprisingly common outcome, especially for a lightweight or infrequently-invoked model, is that a standard CPU-only instance handles the real load adequately, at a fraction of the cost and with none of the quota-planning complexity either accelerator type introduces. Only once a measured CPU-only baseline demonstrates a genuine bottleneck does the GPU-vs-TPU decision (framework support and architectural flexibility vs. large-scale training throughput) actually become the right question to answer next — reaching for acceleration before establishing that baseline repeats this chapter's own "newest/most powerful isn't automatically the right default" lesson from the machine-family discussion.

Summary and What's Next#

This chapter covered Compute Engine end to end: choosing a machine family and custom sizing, images and startup/shutdown scripts, Persistent Disk versus Hyperdisk, OS Login replacing manual SSH key distribution, VM Manager for fleet-wide patching and configuration enforcement, Spot VMs and the specific workload shapes that actually tolerate them safely, managed instance groups as the foundation for a disposable, template-driven fleet, health checks tuned to detect real failure without flapping, and autoscaling policies that combine real metrics with a scheduled floor. Meridian's GPS-ingestion fleet now runs on exactly this pattern — a reliable standard-VM floor, elastic Spot overflow, and automated patching and snapshotting, none of it depending on a human remembering a manual step, and proven out by a real zone outage that resolved itself without a war room.

The specific techniques worth carrying forward: default to the cheapest machine family that measurably fits the workload, never launch an instance on the default service account, treat a MIG's instances as disposable and route every configuration change through the template, and design for interruption tolerance before reaching for a Spot discount rather than after.

Part 5 moves from raw VMs to GKE and serverless compute — where Compute Engine's shared-responsibility burden (OS patching, machine sizing, instance lifecycle) shifts substantially back to Google, and the trade-offs Part 1's shared-responsibility diagram sketched get their full, concrete treatment.