Billing, gcloud CLI & Infrastructure Tooling

Assumes you're comfortable with Part 1's resource hierarchy (organization → folder → project) and the basic gcloud command shape — this chapter goes deep on the billing side of a project and the tooling that turns manual setup into something repeatable.

If you skipped straight to this chapter: the running example throughout this course is Meridian Logistics, a fictional freight-tracking company whose platform team (Priya, Devon, and Ana) you'll see making real decisions in every chapter — Part 1 introduces them and the system architecture diagram this chapter's examples build on.

Table of Contents#

1. What This Chapter Covers
2. Billing Accounts in Depth: Self-Serve vs. Invoiced
3. Linking, Moving, and Unlinking Projects
4. Budgets and Alerts
5. Billing Export and Cost Analysis
6. Multiple Billing Accounts — When and Why to Split
7. Automating a Budget Response, End to End
8. The Infrastructure-as-Code Landscape on GCP
9. Structuring a Terraform Module for a Team, Not Just Yourself
10. Deployment Manager's Deprecation — Why It Matters Even If You Never Used It
11. AI-Assisted Tooling: Gemini CLI, Gemini Cloud Assist, and Application Design Center
12. A Worked Example: Meridian's First Terraform Module
13. Real-World Scenario: The Quarterly Cost Review That Became a Terraform Change
14. Chapter Recap: How the Pieces Connect
15. Pre-Flight Checklist: Is Billing and Tooling Actually Production-Ready?
16. Common Mistakes and Interview Traps
17. Worked Practice Problems
18. Summary and What's Next

What This Chapter Covers#

Part 1 introduced billing accounts and gcloud as concepts you need to bootstrap a project; this chapter goes deep on both — the ACE exam's "managing billing configuration" objective and its "planning and implementing resources using tooling" objective, covered together because in practice a platform team sets up cost controls and infrastructure tooling in the same sitting, not months apart.

🎯 By the end of this chapter, you'll be able to design a billing structure that gives clean per-team cost visibility, configure budget alerts before a workload ever launches, and choose correctly between gcloud, Terraform, Infrastructure Manager, and Config Connector for a given task instead of defaulting to whichever one you already know.

The two halves of this chapter aren't unrelated ACE-syllabus checkboxes bolted together — they're the two sides of the same discipline. Billing visibility (budgets, export, labels) tells you what's actually costing money; infrastructure-as-code tooling is how you act on that finding in a way that stays correct instead of drifting back to the same problem next quarter. The closing worked scenario makes this connection explicit with a real finding that only became a durable fix once it was expressed as a Terraform change, not a one-time manual adjustment.

Billing Accounts in Depth: Self-Serve vs. Invoiced#

A Cloud Billing account is a separate object from a project — it defines who pays, while a project defines what's isolated (Part 1's diagram showed this relationship). GCP offers two billing account types, and which one you have changes both how you pay and what commands are available to you:

# Confirm which type of billing account you're working with —
# the console's "Account type" field, or programmatically:
gcloud billing accounts describe 012345-6789AB-CDEF01

Billing IAM Roles — Separate From Resource IAM#

Billing has its own dedicated IAM roles, scoped to the billing account itself rather than to any project — a genuinely separate permission surface from the compute/storage/IAM roles covered elsewhere in this course, and one of the more commonly-missed distinctions on the ACE exam.

# Grant a project team lead the ability to link new projects to an
# existing billing account, without giving them payment-method access
gcloud billing accounts add-iam-policy-binding 012345-6789AB-CDEF01 \
  --member="user:devon@meridianlogistics.com" \
  --role="roles/billing.user"

From the Trenches: The Card That Expired Mid-Migration#

Three weeks into the GCP migration, Meridian's self-serve billing account's card on file expired without anyone noticing — Google sent an email, which landed in a shared inbox nobody was actively monitoring that week. The billing account moved into a grace period, then resources started facing suspension warnings. The immediate cause was an expired card; the deeper cause was that billing notifications had no owner — they went to an inbox, not to a person or an on-call rotation with an actual response expectation. The fix wasn't just updating the card: Priya registered Essential Contacts (Part 1) for the BILLING category pointed at the platform team's on-call alias, and separately set up a budget alert (this chapter, next section) specifically so a payment failure would surface as a real alert, not an easily-missed email.

Linking, Moving, and Unlinking Projects#

A project has exactly one active billing account link at a time, but that link can change — a project isn't permanently bound to the billing account it was created with.

# Link a project to a billing account (also shown in Part 1)
gcloud billing projects link meridian-shipment-prod \
  --billing-account=012345-6789AB-CDEF01

# Move a project to a DIFFERENT billing account — useful when a
# workload moves from a shared/dev billing account to a dedicated
# production one as it matures
gcloud billing projects link meridian-shipment-prod \
  --billing-account=987654-3210ZY-XWVU98

# Unlink entirely — the project immediately loses access to every
# paid service, falling back to Always Free allowances only
gcloud billing projects unlink meridian-shipment-dev

# Confirm which billing account a project is currently linked to
gcloud billing projects describe meridian-shipment-prod

Realistic Scenario: Re-Linking a Project During an Acquisition#

When a much larger logistics company acquired a smaller regional courier that also ran on GCP, one of Priya's actual integration tasks was re-linking three of the acquired company's projects from their old billing account to Meridian's. The mechanics were a one-line gcloud billing projects link per project — the real work was sequencing it correctly: confirming Meridian's billing account had roles/billing.user granted to Priya first, confirming none of the three projects had an active budget alert that would misfire mid-transition, and doing the re-link during a low-traffic maintenance window in case any billing-dependent quota briefly reset during the switch. The command is trivial; the change-management around a production billing re-link is where the real risk lives — the same lesson as the earlier region-lock exception scenario in Part 1: a simple, well-understood command can still carry real operational risk if the surrounding process isn't deliberate.

Budgets and Alerts#

A budget is a spending threshold with configurable alert rules, scoped either to a whole billing account or to specific projects within it — and critically, a budget on its own only alerts, it doesn't stop spending unless you separately wire it to automation that reacts to the alert.

# Create a budget scoped to Meridian's production project, alerting
# at 50%, 90%, and 100% of a $5,000 monthly target (Cloud Billing's
# own defaults — worth keeping unless you have a specific reason not to)
gcloud billing budgets create \
  --billing-account=012345-6789AB-CDEF01 \
  --display-name="Meridian Shipment Prod — Monthly" \
  --budget-amount=5000USD \
  --filter-projects=projects/meridian-shipment-prod \
  --threshold-rule=percent=0.5 \
  --threshold-rule=percent=0.9 \
  --threshold-rule=percent=1.0

Three separate notification paths exist, and picking the right one for your team's actual workflow matters more than picking all three by default:

Only the Pub/Sub path lets you build real automated response — email and Monitoring notifications are informational only.

Worked Example: Setting a Budget That Reflects Real Growth, Not a Guess#

Ana set Meridian's first production budget at a flat $5,000/month, picked somewhat arbitrarily during initial setup. Four months later, actual spend had grown from $2,100 to $4,600 as vehicle count grew — meaning the 90% alert (at $4,500) had already fired twice in the prior month, both times for expected growth, not a problem. A budget that fires routinely for normal growth teaches the team to ignore it, which is worse than having no budget at all — the alert stops functioning as a signal the moment "it always fires" becomes the expected state.

The fix was recalculating the budget against Meridian's actual growth curve rather than a flat number: a forecasted-spend threshold rule (alerting when Cloud Billing's own forecast — not just current spend — projects crossing a limit) combined with a quarterly budget-amount review tied to the same vehicle-count growth plan already driving the Part 1 quota-increase worked example. The two numbers (compute quota headroom and budget ceiling) should move together, since they're driven by the exact same growth curve.

# A forecasted-spend threshold rule alerts on a PROJECTION crossing
# the limit, not just actual spend already crossing it — genuinely
# earlier warning than a percent-of-current-spend rule alone
gcloud billing budgets create \
  --billing-account=012345-6789AB-CDEF01 \
  --display-name="Meridian Shipment Prod — Monthly (Q3 revision)" \
  --budget-amount=6500USD \
  --filter-projects=projects/meridian-shipment-prod \
  --threshold-rule=percent=0.9,basis=forecasted-spend \
  --threshold-rule=percent=1.0,basis=current-spend

A Preview of Cost Optimization: Committed and Sustained Use Discounts#

Two discount mechanisms are worth knowing exist even though this course's FinOps depth lives in Course 3 (GCP SRE & Observability): sustained use discounts apply automatically to Compute Engine workloads that run a large fraction of the billing month, no commitment required; committed use discounts (CUDs) trade a 1- or 3-year spend commitment for a steeper discount, appropriate once a workload's baseline size is genuinely stable and predictable. Meridian hasn't committed to either yet — the GPS-ingestion fleet is still growing quarter over quarter, and locking in a 1-year CUD against a size that will be wrong in three months would trade flexibility for a discount that doesn't actually net out ahead.

Realistic Scenario: The Load Test That Nearly Doubled the Monthly Bill#

Ana ran a load test against a staging Cloud Run service, intentionally scaling it to simulate 10x normal traffic — and forgot to scale it back down afterward, leaving min-instances set high for the following four days over a weekend. The staging project's 90%-of-budget alert fired on Saturday morning; Ana's on-call rotation picked it up within twenty minutes and rolled the setting back, catching what would otherwise have been roughly four extra days of over-provisioned compute before it silently became a full month's worth. The budget alert didn't prevent the mistake — nothing stops a human from misconfiguring min-instances — but it caught the consequence fast enough that the actual cost impact was a few hundred dollars, not a few thousand.

Billing Export and Cost Analysis#

Billing export sends every line item of your bill to BigQuery in near-real-time, turning "what did we spend on last month" from a console-report question into a SQL query you can run, chart, or automate against.

# Configure standard usage cost export to a BigQuery dataset —
# console-driven setup for the initial link, but once configured,
# every subsequent day's costs land there automatically
bq mk --dataset meridian-shared-logging:billing_export

# A representative query once export is flowing: cost by label,
# last 30 days — this is exactly how Meridian answers "which
# team's workloads are driving the bill" without waiting on finance

SELECT
  labels.value AS team,
  SUM(cost) AS total_cost
FROM `meridian-shared-logging.billing_export.gcp_billing_export_v1`,
  UNNEST(labels) AS labels
WHERE labels.key = 'team'
  AND usage_start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
GROUP BY team
ORDER BY total_cost DESC

This is exactly why Part 1's insistence on labeling every resource with team, env, and cost-center matters beyond the console filtering it also enables — the billing export table only has something meaningful to GROUP BY if the underlying resources actually carried the labels in the first place. An unlabeled fleet exports perfectly clean cost data with no way to attribute any of it.

Scheduled Cost Reports — Making the Query a Habit, Not a One-Off#

A cost-attribution query that only runs when someone remembers to run it manually eventually stops running. BigQuery's scheduled queries turn Ana's cost-by-team query into a standing weekly report that lands in a Slack channel without anyone re-running it by hand:

# Schedule the same team-attribution query from earlier to run every
# Monday morning, writing results to a dedicated reporting table
bq mk --transfer_config \
  --project_id=meridian-shared-logging \
  --data_source=scheduled_query \
  --target_dataset=billing_export \
  --display_name="Weekly cost by team" \
  --schedule="every monday 08:00" \
  --params='{"query":"SELECT labels.value AS team, SUM(cost) AS total_cost FROM `meridian-shared-logging.billing_export.gcp_billing_export_v1`, UNNEST(labels) AS labels WHERE labels.key = \"team\" AND usage_start_time >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 7 DAY) GROUP BY team ORDER BY total_cost DESC","destination_table_name_template":"weekly_cost_by_team","write_disposition":"WRITE_TRUNCATE"}'

Where to Actually Check Current Spend#

Three different surfaces all show cost data, and picking the wrong one for the task wastes time:

Ana's team defaults to the BigQuery export path for anything asked more than once, and the console only for a genuine one-off "let me just look at this right now" check — the same reasoning Part 1 applied to choosing between the console, gcloud, and Terraform for infrastructure changes, applied here to cost visibility instead.

Multiple Billing Accounts — When and Why to Split#

Most companies run everything through one billing account for years; splitting into multiple genuinely makes sense in a specific, narrow set of situations rather than as a default:

Meridian doesn't split — one billing account, cost attribution handled entirely through labels and folder structure, which the decision table above suggests is the right call for a single-entity company with no exotic payment-terms need. Splitting billing accounts to solve a cost-attribution problem that labels already solve is unnecessary overhead — it adds a second thing (which billing account is a project linked to) that has to stay correct, for no benefit a well-labeled single account doesn't already provide.

From the Trenches: The Acquisition That Left Two Billing Accounts Running#

After the courier acquisition mentioned earlier in this chapter, Meridian briefly ran with the acquired company's original billing account still active in parallel with Meridian's own, "temporarily," while integration work continued. Eight months later, finance discovered the "temporary" second account was still live, still being invoiced separately, and had drifted from having any dedicated owner — the original team's finance contact had left the company during the transition. The immediate cause was an integration task left in an intentionally temporary state; the deeper cause was that "temporary" had no expiry date or owner attached to it, the same pattern as the earlier region-lock policy exception in Part 1. Every deliberately temporary state — a billing account, a policy exception, a stopgap IAM grant — needs an explicit owner and a real date it gets revisited, or it quietly becomes permanent.

Automating a Budget Response, End to End#

The Pub/Sub notification path teased earlier in the diagram is worth showing fully worked, because "wire it to automation" is easy to say and genuinely useful to see built once. This is the actual pattern for a hard-stop safety net on a non-critical environment — deliberately not used on meridian-shipment-prod, where an automatic shutdown would itself be an outage, but exactly right for meridian-shipment-dev, where nothing customer-facing depends on uptime.

The function itself decides whether to act — the budget and Pub/Sub trigger are identical for every project; the response logic is what makes production safe from an accidental shutdown.

# main.py — Cloud Function triggered by the budget's Pub/Sub topic.
# Deliberately conservative: only ever acts on the one project this
# is explicitly scoped to, never a wildcard match.
import base64
import json
from google.cloud import billing_v1

PROTECTED_DEV_PROJECT = "meridian-shipment-dev"

def disable_dev_billing(event, context):
    # The budget notification's Pub/Sub message body carries budgetDisplayName,
    # costAmount, budgetAmount, and alertThresholdExceeded — not a project ID
    # directly, which is exactly why the budget's own display name is the
    # deliberate, explicit match key here rather than something inferred.
    payload = json.loads(base64.b64decode(event["data"]).decode("utf-8"))
    budget_name = payload.get("budgetDisplayName", "")

    if PROTECTED_DEV_PROJECT not in budget_name:
        print(f"Alert for budget '{budget_name}' — not the protected dev project, no action taken.")
        return

    if payload.get("alertThresholdExceeded", 0) < 1.1:
        print("Under the hard-stop threshold — alert only, no action.")
        return

    client = billing_v1.CloudBillingClient()
    name = f"projects/{PROTECTED_DEV_PROJECT}"
    client.update_project_billing_info(
        name=name,
        project_billing_info={"billing_account_name": ""},
    )
    print(f"Billing unlinked for {PROTECTED_DEV_PROJECT} — hard spend limit exceeded.")

Detecting Drift Between Terraform State and Reality#

A terraform plan run on a schedule — not just before a deliberate change — is one of the cheapest ways to catch when a resource has quietly diverged from what Terraform believes it manages, usually because someone made a manual console edit "just this once" to fix something urgently.

# .github/workflows/drift-detection.yml — a scheduled CI job that runs
# plan (never apply) and alerts if it detects any unexpected diff
name: Terraform Drift Detection
on:
  schedule:
    - cron: "0 6 * * *"  # every day at 06:00 UTC
jobs:
  detect-drift:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: hashicorp/setup-terraform@v3
      - run: terraform init
      - run: terraform plan -detailed-exitcode
        id: plan
        continue-on-error: true
      - name: Alert on drift
        if: steps.plan.outputs.exitcode == '2'
        run: echo "Drift detected — Terraform plan shows unexpected changes" # wire to Slack/PagerDuty

terraform plan -detailed-exitcode returns exit code 2 specifically when it detects a difference between state and real infrastructure (as opposed to 0 for no changes, 1 for an error) — the exact signal a drift-detection job needs to distinguish "everything matches" from "something changed outside Terraform." Meridian's platform team treats a drift alert the same as any other production alert: someone investigates why the resource changed before deciding whether to apply (bringing reality back to match Terraform) or update the Terraform config (if the manual change was actually the correct new desired state).

Negotiated Discounts vs. Committed Use Discounts — Different Levers#

Beyond the CUD/sustained-use preview earlier in this chapter, a company crossing a certain spend threshold can also negotiate directly with a Google account team for custom pricing — a genuinely different lever from either discount mechanism, worth distinguishing clearly:

Meridian is years away from the third option — negotiated agreements are typically reserved for customers spending well into six or seven figures annually — but Priya tracks it as a real future milestone tied to the same growth curve driving this chapter's quota and budget examples, not a hypothetical she's ignoring indefinitely.

The Infrastructure-as-Code Landscape on GCP#

GCP has more first-party infrastructure-as-code options than most clouds, and picking the wrong one for a given layer is a real, recurring source of team friction.

Terraform stays the same regardless of who runs it — Infrastructure Manager is Google's own managed runner for the exact same Terraform configs, not a different language.

Fabric FAST — Not Reinventing the Landing Zone#

Part 1's whole "bootstrap a landing zone" sequence — folders, org policies, shared-services projects, audit logging — is exactly the kind of foundational work Google's own Fabric FAST (Fabric being the umbrella name for Google's published Terraform blueprint modules) exists to give you a tested starting point for, instead of hand-writing every module from a blank file.

# Fabric FAST ships as a cloned repository of composable Terraform
# stages, applied in order — bootstrap, resource hierarchy, security,
# networking, and per-team/project stages layered on top
git clone https://github.com/GoogleCloudPlatform/cloud-foundation-fabric.git
cd cloud-foundation-fabric/fast/stages/0-bootstrap
terraform init && terraform plan

Meridian didn't adopt Fabric FAST wholesale — a 12-person company's landing zone is simple enough that hand-written Terraform (like Part 1's bootstrap sequence and this chapter's Pub/Sub module) stays easy to reason about end to end. Fabric FAST earns its complexity budget at a materially larger scale: dozens of teams, a real platform-engineering function maintaining shared modules, and enough landing-zone surface area that reinventing it from scratch would take real engineer-months. Knowing it exists — and the specific pain threshold where reaching for it starts paying off — matters more at this stage than a company Meridian's size actually adopting it.

From the Trenches: The Terraform State Fight With Config Connector#

A team piloting Config Connector let application engineers manage their own Cloud SQL instances via Kubernetes manifests — including one instance Terraform had already created and was still managing in its own state file. Both systems believed they owned that resource's configuration; the first time an application engineer edited the Kubernetes manifest, Config Connector's reconciler changed a setting Terraform's next apply immediately reverted, and vice versa, in a slow-motion fight neither side "won." The immediate cause was one resource claimed by two systems; the deeper cause was skipping the explicit layer-ownership decision from the tip above before adopting a second IaC tool. The fix was a hard rule: Terraform-created resources are annotated as such and never imported into Config Connector, full stop — a resource belongs to exactly one system of record, decided before either tool touches it, not discovered after they conflict.

Structuring a Terraform Module for a Team, Not Just Yourself#

The single-file main.tf shown above works for a demo; a module more than one person maintains needs real structure so a reviewer can find what changed without reading every line:

# variables.tf — every input documented with a type and description,
# so a reviewer never has to guess what a bare value means
variable "environment" {
  description = "Deployment environment (prod, staging, dev)"
  type        = string
  validation {
    condition     = contains(["prod", "staging", "dev"], var.environment)
    error_message = "environment must be one of: prod, staging, dev."
  }
}

variable "cost_center" {
  description = "Cost-center label applied to every resource in this module"
  type        = string
}

# Apply the same module against different environments using
# separate tfvars files — one module, parameterized, instead of
# copy-pasted per environment
terraform apply -var-file=environments/prod.tfvars

IaC Terminology Map#

The same cross-provider translation habit from Part 1 applies to infrastructure tooling too:

The one place this mapping breaks down: AWS deliberately keeps investing in CloudFormation as a first-party option alongside Terraform, while GCP has moved its own native tool (Deployment Manager) toward deprecation in favor of managing Terraform itself (Infrastructure Manager) rather than maintaining a separate proprietary format — a real difference in each provider's IaC strategy, not just a naming difference.

Infrastructure Manager in Practice — the Same Terraform, a Managed Runner#

Infrastructure Manager doesn't ask you to learn a new language — it takes the exact same Terraform configuration shown throughout this chapter and runs it through a managed Cloud Build execution, storing state in a Google-managed Cloud Storage location instead of one you provision and secure yourself:

# Deploy the same gps-ingestion module through Infrastructure Manager
# instead of running terraform apply locally — Google manages the
# execution and the state backend
gcloud infra-manager deployments apply projects/meridian-shipment-prod/locations/us-central1/deployments/gps-ingestion \
  --service-account=projects/meridian-shipment-prod/serviceAccounts/infra-manager@meridian-shipment-prod.iam.gserviceaccount.com \
  --local-source=./gps-ingestion

The trade-off is real, not one-sided: Infrastructure Manager removes the "who operates the Terraform runner and state backend" question entirely, at the cost of Google controlling exactly how and where that execution happens — a fit for a team that wants Terraform's declarative model without also owning a CI/CD pipeline dedicated to running it. Meridian's platform team, already comfortable running Terraform through a GitHub Actions pipeline it controls (the same drift-detection workflow shown later in this chapter), hasn't adopted Infrastructure Manager — the trade-off doesn't net out in their favor when the CI/CD pipeline they'd need anyway for review/approval gates already exists.

Deployment Manager's Deprecation — Why It Matters Even If You Never Used It#

Google deprecated Deployment Manager (its original, GCP-native IaC tool) with support ending April 1, 2026 and full service turn-down after June 30, 2027 — Infrastructure Manager is the named migration path. This matters for two reasons even to a team that never adopted Deployment Manager directly: first, any inherited legacy configuration (an acquired company's old landing zone, a years-old internal tool) may still depend on it and needs a migration plan on a real deadline, not indefinitely; second, it's a useful, concrete example of how GCP's own first-party tooling landscape shifts over time — the "right" tool for a given IaC layer is worth re-checking periodically, not something a team decides once and never revisits.

# Google publishes a conversion tool for exactly this migration —
# DM Convert translates Deployment Manager configs toward Terraform
dm-convert convert --config=legacy-deployment.yaml --output-dir=./terraform-migrated

• Inventory every Deployment Manager config still in active use (gcloud deployment-manager deployments list --project=<id> across every project)
• Run dm-convert against each one and diff the generated Terraform against the original config's actual behavior, not just its syntax
• Import existing real resources into the converted Terraform state (terraform import) rather than letting apply try to recreate them
• Decommission the Deployment Manager deployment only after the Terraform equivalent has been verified against production, not before

AI-Assisted Tooling: Gemini CLI, Gemini Cloud Assist, and Application Design Center#

Gemini Cloud Assist and Application Design Center, integrated via Gemini CLI, turn a natural-language description of an application into a visual architecture diagram plus production-ready Terraform, gcloud, or kubectl output — genuinely useful for accelerating the first draft of an environment's infrastructure, not a replacement for understanding what that output actually provisions.

# Gemini CLI, with the Cloud Assist/Application Design Center MCP
# integration configured, can turn a description into a design +
# deployable blueprint — output reviewed like any other generated
# code before it's applied to a real environment
gemini "Design a Cloud Run service behind a load balancer, backed by
a Cloud SQL Postgres instance, for a shipment-tracking API expecting
around 200 requests per second at peak"

The decision framework that matters here isn't "AI tool vs. no AI tool" — it's what to do with the output:

From the Trenches: The Generated Module That Skipped Encryption#

Devon used Gemini CLI to draft a first-pass Terraform module for a new Cloud Storage bucket during a time-crunched sprint, reviewed the plan output for the resource names and settings he was actively thinking about (bucket name, location, lifecycle rule), applied it, and moved on. Three weeks later a security review flagged that the bucket had no customer-managed encryption key (CMEK) configured — the generated module used GCP's default encryption, technically secure but short of the CMEK-everywhere standard Meridian's SOC 2 scope required, a requirement Devon knew but wasn't actively scanning for in that specific review because he was focused on the settings he'd asked for, not auditing everything the tool added on its own. The lesson wasn't "don't use AI-assisted generation" — it was that reviewing generated infrastructure against your own checklist (the same Pre-Flight Checklist pattern this course uses at the end of several chapters) catches gaps a glance at "does this look reasonable" won't, precisely because a fast first draft satisfies the request you typed, not the standard you didn't think to restate.

A Worked Example: Meridian's First Terraform Module#

Putting Terraform's actual usage on GCP together — this is the real module Devon wrote to provision the GPS-ingestion Pub/Sub topic and its dead-letter counterpart, the first piece of infrastructure Meridian moved from a manual gcloud command into version control:

# main.tf — Pub/Sub topic + dead-letter topic for GPS ingestion
terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 6.0"
    }
  }
}

provider "google" {
  project = "meridian-shipment-prod"
  region  = "us-central1"
}

resource "google_pubsub_topic" "gps_pings" {
  name   = "gps-pings"
  labels = {
    team        = "platform"
    env         = "prod"
    cost-center = "logistics-ops"
  }
}

resource "google_pubsub_topic" "gps_pings_dlq" {
  name   = "gps-pings-dlq"
  labels = {
    team        = "platform"
    env         = "prod"
    cost-center = "logistics-ops"
  }
}

resource "google_pubsub_subscription" "gps_ingestion_worker" {
  name  = "gps-ingestion-worker"
  topic = google_pubsub_topic.gps_pings.name

  dead_letter_policy {
    dead_letter_topic     = google_pubsub_topic.gps_pings_dlq.id
    max_delivery_attempts = 5
  }

  ack_deadline_seconds = 30
}

# The standard Terraform loop — every change reviewed as a diff
# before it touches real infrastructure
terraform init
terraform plan -out=tfplan
terraform apply tfplan

Notice the same labels from Part 1 applied directly in the Terraform resource itself — that's deliberate: labeling belongs in the IaC definition, not as a manual after-the-fact console edit, so every future terraform apply re-asserts the correct labels instead of a human having to remember to re-add them if a resource is ever recreated.

Remote State — Why Local State Is a Trap From Day One#

The module above is intentionally shown without a backend block first, because the natural next mistake is running it exactly as-is and letting Terraform write terraform.tfstate to a local laptop disk. State on a laptop means only that laptop can safely run terraform apply, and a lost or wiped laptop means losing the only record of what Terraform thinks exists — a real, avoidable single point of failure for a team of more than one person.

# backend.tf — GCS-backed remote state, shared across the whole
# platform team, with built-in state locking so two people can't
# apply conflicting changes at the same time
terraform {
  backend "gcs" {
    bucket = "meridian-shared-terraform-state"
    prefix = "gps-ingestion/prod"
  }
}

# Migrating from local to remote state on an already-applied module —
# Terraform detects the backend change and offers to copy state over
terraform init -migrate-state

How Terraform Actually Authenticates to GCP#

The provider "google" block in the earlier module didn't include any credentials — that's not an oversight, it's Terraform relying on Application Default Credentials (ADC), the same credential-resolution chain most GCP client libraries and tools use rather than a Terraform-specific mechanism.

The metadata-server path is why Terraform running inside Cloud Build or on a GCE VM needs no explicit credential configuration at all — Part 3 covers exactly how that attached-identity mechanism works underneath.

# The developer-machine path: authenticate once, Terraform (and most
# gcloud client libraries) picks it up automatically afterward
gcloud auth application-default login

# The CI/CD path (Cloud Build, GitHub Actions, etc.): a dedicated
# service account, never a human's personal credentials
export GOOGLE_APPLICATION_CREDENTIALS=/secrets/terraform-sa-key.json

Who Can Read Terraform State — and Why That Matters#

A Terraform state file is not just a list of resource IDs — it can contain sensitive values in plain text, including database passwords and API keys set via a resource's arguments, because Terraform needs the full resource configuration to compute future diffs. Treat the state bucket's IAM as seriously as you'd treat a secrets manager, not as ordinary infrastructure metadata:

# Restrict the state bucket to exactly the identities that need it —
# the platform team's own accounts and the CI/CD service account,
# nothing broader
gcloud storage buckets add-iam-policy-binding \
  gs://meridian-shared-terraform-state \
  --member="group:platform-team@meridianlogistics.com" \
  --role="roles/storage.objectAdmin"

From the Trenches: Two Applies at the Same Time#

Before Meridian moved to remote state with locking, Devon and a contractor both ran terraform apply against the same local-state configuration within minutes of each other during a rushed fix, each working from their own stale copy of terraform.tfstate. The contractor's apply completed first; Devon's apply — still reasoning from a state file that didn't know about the contractor's change — then tried to "fix" what looked like drift, deleting a subscription the contractor had just created. The immediate cause was two people applying from two different state snapshots; the deeper cause was no locking mechanism existed to make that literally impossible. GCS-backed remote state's built-in locking (via a lease on the state object) means a second concurrent apply now fails fast with a clear "state is locked" error instead of silently proceeding against stale data.

Real-World Scenario: The Quarterly Cost Review That Became a Terraform Change#

Meridian's first quarterly cost review (billing export query, exactly the pattern shown earlier in this chapter) surfaced something specific: the GPS-ingestion Compute Engine fleet was consistently running at low overnight utilization — expected, since most of Meridian's fleet operates during business hours — but the managed instance group's min-instances setting kept the full daytime-sized fleet running 24 hours a day regardless.

The finding came from a SQL query against exported billing data; the fix belonged in Terraform, not a manual console click, because manually reducing min-instances for one night would just need to be manually reverted the next morning and would drift the moment anyone forgot:

# Before: static, no time-awareness — a comfortable default when the
# fleet was small, expensive once traffic patterns became predictable
resource "google_compute_region_autoscaler" "gps_worker" {
  name   = "gps-worker-autoscaler"
  region = "us-central1"
  target = google_compute_region_instance_group_manager.gps_worker.id

  autoscaling_policy {
    min_replicas = 24
    max_replicas = 48
    cpu_utilization {
      target = 0.6
    }
  }
}

# After: a lower off-hours floor, driven by a schedule-based policy —
# Part 4 covers autoscaling policies in full depth; this preview shows
# the change that came directly out of the billing-export finding above
resource "google_compute_region_autoscaler" "gps_worker" {
  name   = "gps-worker-autoscaler"
  region = "us-central1"
  target = google_compute_region_instance_group_manager.gps_worker.id

  autoscaling_policy {
    min_replicas = 8
    max_replicas = 48
    cpu_utilization {
      target = 0.6
    }
    scaling_schedules {
      name                  = "business-hours-floor"
      min_required_replicas = 24
      schedule              = "0 7 * * 1-5"
      time_zone             = "America/Chicago"
      duration_sec          = 39600
    }
  }
}

This is the actual loop this chapter's two halves (billing visibility and IaC discipline) are meant to close together: a cost finding with no IaC path to act on it stays a finding forever; IaC with no cost visibility feeding it optimizes blind. Neither half is complete without the other.

Chapter Recap: How the Pieces Connect#

A reminder: mindmap diagrams stay in plain default style deliberately — mermaid's mindmap renderer doesn't reliably support classDef coloring, so vary shape/emphasis through the labels themselves rather than color here.

Pre-Flight Checklist: Is Billing and Tooling Actually Production-Ready?#

• At least one budget alert configured on every project, sized against actual growth data, not a guess
• Budget notifications routed to a channel with a real, guaranteed human response (on-call paging), not email alone
• Essential Contacts registered for the BILLING category, pointed at a monitored alias
• Billing export flowing to BigQuery, and every billable resource carrying the labels the export query depends on
• Billing IAM roles granted at the narrowest level needed (billing.user for linking, not billing.admin, unless payment-method access is genuinely required)
• Foundational infrastructure defined in Terraform, not created by hand via gcloud or the console
• Terraform state stored remotely (GCS backend) with locking, never on a single person's laptop
• No new infrastructure being built on Deployment Manager

Common Mistakes and Interview Traps#

Worked Practice Problems#

1. Meridian's finance team asks whether splitting into two billing accounts (one per environment: production and non-production) would improve cost visibility. Is this the right move, and what should Priya recommend instead?

No — splitting billing accounts is the wrong tool for this specific problem. Cost visibility by environment is already fully achievable through the env label already applied to every resource, queried via billing export (this chapter's SQL example) or filtered directly in the Cloud Billing console. Splitting billing accounts adds real operational overhead (a second account to manage, a second place a project's billing link can silently point to the wrong account) without solving a problem labels don't already solve. The decision table in this chapter reserves billing-account splits for genuinely different legal entities, payment terms, or hard-isolation requirements — none of which apply here.

2. A budget's 100% threshold alert fired via email last night, but nobody noticed until this morning, and spending kept climbing overnight. What's the actual gap, and what two changes close it?

The gap isn't the budget itself — it correctly fired — it's that the only configured notification path (email) has no guaranteed human response time attached to it. Two changes close it: route the same budget alert through a Cloud Monitoring notification channel wired into the team's existing on-call paging (so it interrupts someone, not just sits in an inbox), and consider whether a Pub/Sub-triggered automated response (disabling a specific non-critical project's billing, say) is warranted for that specific budget's severity — turning "someone eventually reads an email" into either "someone is paged immediately" or "the spend stops automatically."

3. Devon wants to let each application team provision their own Pub/Sub topics and Cloud Storage buckets via Config Connector manifests alongside their Kubernetes deployments, while the platform team keeps Terraform for networking and IAM. What's the one rule that keeps this from becoming the state-conflict problem this chapter described, and why does it have to be decided before adoption, not after a conflict appears?

The rule: a resource belongs to exactly one IaC system of record, decided by which layer it sits in — networking/IAM stays in Terraform, application-level resources (topics, buckets) go to Config Connector, and neither tool is ever pointed at a resource the other already manages. It has to be decided before adoption because once both systems believe they own the same resource's configuration, each apply/reconcile cycle fights the other's last change — a slow-motion conflict that's confusing to debug precisely because neither tool reports an error, they just keep silently reverting each other's edits.

4. Devon set up Terraform with a local state file six months ago, and the platform team has since grown from one person to three. What's the actual risk today that wasn't a risk when Devon was working alone, and what's the fix?

With one person, local state has an implicit, unstated locking mechanism: only one person is ever running apply, so there's no possibility of a concurrent conflicting change. With three people, that implicit safety disappears — any two people applying around the same time can each be reasoning from a stale local copy of state, and Terraform has no way to detect or prevent that without a shared backend. The fix is migrating to a GCS-backed remote state (terraform init -migrate-state), which adds real state locking — a second concurrent apply fails fast with an explicit lock error instead of silently applying against outdated state, exactly the failure Meridian actually hit and used to justify the migration.

5. A daily scheduled terraform plan drift-detection job starts reporting drift on a resource every single day, always the same three-line diff. What does this recurring (rather than one-time) pattern most likely indicate, and how does it differ from a genuine one-off manual change?

A one-off manual change shows up as drift once, and running apply (or updating the Terraform config to match, if the manual change was correct) makes the drift alert stop permanently. A recurring identical diff every single day instead points to something outside Terraform's control continuously re-changing that same field — commonly an autoscaler, a GCP-managed default that gets silently re-applied, or another automated system (a startup script, a separate tool) fighting over the same field Terraform manages. The fix isn't repeatedly re-applying the same plan; it's identifying the other actor changing that field and either excluding the field from Terraform's management (via a lifecycle { ignore_changes = [...] } block, if the other system is legitimately meant to own it) or removing the conflicting automation, the same "one resource, one system of record" principle from the Config Connector scenario earlier in this chapter.

Summary and What's Next#

This chapter completed the operational foundation Part 1 started: real billing account mechanics (self-serve vs. invoiced, linking/unlinking, budgets that actually reach a human, cost export that makes labels queryable) and the IaC tooling landscape (Terraform as the default, Infrastructure Manager as its managed runner, Config Connector for a Kubernetes-native application layer, and Deployment Manager's firm deprecation timeline). Meridian's landing zone from Part 1 is now something that could, in principle, be torn down and rebuilt identically from Terraform state — the actual bar for "this environment is real infrastructure, not a pile of manual gcloud commands," and the standard every later chapter's own worked examples build on without re-explaining it.

The specific techniques worth carrying forward, even outside the ACE exam's own scope: label everything before it launches, wire budget alerts to a real human response instead of an inbox, put every foundational resource in version-controlled Terraform with remote state, and check cost visibility and infrastructure changes against each other regularly rather than treating them as two separate teams' separate concerns.

Part 3 moves from what exists to who can touch it — a full deep dive into IAM: principals, roles, bindings, service accounts, impersonation, and the Workload Identity Federation mechanism that lets a CI/CD pipeline or an external workload authenticate to GCP without ever holding a long-lived key file.