Infrastructure as Code & GitOps

Table of Contents#

1. Why Infrastructure Needs the Same Discipline as Application Code
2. Terraform on GCP: Provider, State, and Modules
3. Remote State: Why a Laptop's Local .tfstate Is a Production Incident Waiting to Happen
4. Cloud Foundation Toolkit and Fabric FAST: Google's Own Blueprints
5. Infrastructure Manager: Google's Managed Terraform Runner
6. Config Connector: Kubernetes-Native GCP Resource Management
7. GitOps: The Delivery Model Underneath All of This
8. Config Sync: GitOps for What Runs Inside the Cluster
9. Choosing Between Terraform, Config Connector, and Infrastructure Manager
10. A Full Worked Example: Meridian's Environment Factory Module
11. Common Mistakes and Interview Traps
12. Worked Practice Problems
13. Summary and What's Next

Why Infrastructure Needs the Same Discipline as Application Code#

A resource created by hand through the Console or a one-off gcloud command has no record of why it exists, no review before it changed, and no way to reproduce it if the project is ever deleted. Infrastructure as Code (IaC) closes that gap by making infrastructure changes a pull request instead of a click — reviewed, versioned, and reapplied identically from a clean state. The PCDE exam guide lists this under "managing infrastructure" (section 1.2) and names the specific GCP tooling: Infrastructure Manager, Cloud Foundation Toolkit, Config Connector, and GitOps as the delivery model tying them together.

Part 1 established where infrastructure lives — the tooling project pattern, environment-specific policy. This chapter is about how that infrastructure gets defined and kept in sync with what's actually deployed, which is the harder and more durable half of the problem: a resource hierarchy is drawn once, but the resources inside it change every week for the life of the platform.

Meridian's platform team hit this gap directly: their meridian-dev, meridian-staging, and meridian-prod projects from Part 1 were created by hand, following the gcloud commands in that chapter — accurate today, but with no record anywhere of why meridian-prod has a stricter compute IP policy than meridian-dev, and no way to recreate that exact configuration if Priya needed a fourth environment next quarter for a new product line.

What to notice: the risk isn't that hand-run commands are slower — it's that they leave no artifact behind for the next engineer, the next audit, or the next disaster-recovery drill to reason about.

Terraform on GCP: Provider, State, and Modules#

Terraform is the dominant IaC tool across all three major clouds, and GCP's own google provider (with a narrower google-beta provider for preview features) is the mechanism translating Terraform's declarative HCL into actual API calls against GCP's control plane. If you've used Terraform on AWS or Azure, the mental model transfers directly — what changes is only the resource type names and the shape of each provider's arguments.

# A minimal, real Terraform configuration creating one of Meridian's
# environment projects — the SAME resource Part 1's gcloud command
# created by hand, now declared, reviewable, and reproducible.
terraform {
  required_providers {
    google = {
      source  = "hashicorp/google"
      version = "~> 6.0"
    }
  }
}

resource "google_project" "environment" {
  name            = "Meridian ${title(var.environment_name)}"
  project_id      = "meridian-${var.environment_name}"
  folder_id       = var.environment_folder_id
  billing_account = var.billing_account_id

  labels = {
    environment = var.environment_name
    managed_by  = "terraform"
  }
}

resource "google_project_service" "required_apis" {
  for_each = toset([
    "compute.googleapis.com",
    "container.googleapis.com",
    "run.googleapis.com",
  ])
  project = google_project.environment.project_id
  service = each.value
}

⚙️ The mechanism worth internalizing: terraform plan doesn't guess — it calls the same GCP APIs the Console does to read current state, diffs that against what your .tf files declare, and shows you exactly what would change before anything actually changes. This preview step is the single biggest reason IaC beats hand-run commands for anything touching production: a reviewer can catch "this would delete the production Cloud SQL instance" in a pull request, long before terraform apply ever runs.

Remote State: Why a Laptop's Local .tfstate Is a Production Incident Waiting to Happen#

Terraform's state file is the thing that makes the plan/apply preview possible — it's Terraform's own record of what it last created, mapping every resource block in your .tf files to a real, live GCP resource ID. Left as the default local terraform.tfstate file, state lives on whichever single machine ran terraform apply last. That's fine for a solo experiment; it's a genuine production risk the moment a second engineer or a CI pipeline needs to run terraform apply too.

🔍 From the Trenches: A two-person team ran Terraform from local state for eight months without incident, because in practice only one of them ever ran apply. When the second engineer finally ran terraform apply from her own laptop for an urgent fix, Terraform had no record of the first engineer's most recent changes — her local state file was stale by three weeks. Terraform proposed to recreate eleven resources it believed didn't exist yet, including a Cloud SQL instance actually holding production data. She caught it because the plan output looked implausibly large for a two-line config change, not because anything technical stopped her — a smaller, more routine change would have gone through. The underlying condition wasn't "she made a mistake" — it was that the team's workflow had no mechanism forcing state to be shared and locked at all, so the danger sat there silently for eight months before the second apply ever happened to expose it.

The fix is a remote backend — for GCP, almost always a GCS bucket with object versioning and Terraform's native state locking:

terraform {
  backend "gcs" {
    bucket = "meridian-terraform-state"
    prefix = "environments/staging"
  }
}

Cloud Foundation Toolkit and Fabric FAST: Google's Own Blueprints#

Part 1 introduced these by name; here's what actually using one looks like. Cloud Foundation Toolkit (CFT) is a library of individually reusable, versioned Terraform modules (terraform-google-modules on GitHub) — a project-factory module, a network module, an iam module — each one independently useful without adopting the whole opinionated stack.

Fabric FAST, built by Google Cloud's own Professional Services organization, goes further: it's a complete, opinionated multi-stage Terraform pipeline, not just a module library. FAST organizes an organization's bootstrap into separate "stages" — each its own Terraform root module — that hand off outputs to the next stage via a defined contract: a bootstrap stage sets up the org-level Terraform service accounts and state bucket, a resource-management stage builds the folder hierarchy, a networking stage builds shared VPC, and so on. Each stage can be run and reasoned about independently, but together they produce a complete landing zone.

What to notice: every stage after Bootstrap consumes outputs from an earlier stage rather than hardcoding values — Fabric FAST's actual engineering contribution is this staged handoff design, not any single module in isolation.

Infrastructure Manager: Google's Managed Terraform Runner#

Everything so far describes what Terraform configuration to write — Infrastructure Manager answers a different question: where does terraform apply actually run, and who operates that runner? Without it, a team runs Terraform from a CI pipeline they build and maintain themselves (Cloud Build, GitHub Actions, Jenkins — Part 3 covers Cloud Build specifically), managing the runner's permissions, its state backend wiring, and its plan/apply approval flow by hand.

Infrastructure Manager (Infra Manager) is Google's own managed execution environment for that same Terraform configuration — it calls a Terraform configuration a "deployment," runs plan and apply inside a Google-managed environment via Cloud Build under the hood, and stores state in a Google-managed GCS location automatically, removing the backend-bootstrap chicken-and-egg problem from the previous section entirely.

# Create an Infrastructure Manager deployment pointing at a Git repo
# holding the Terraform config from earlier in this chapter — Google
# runs plan/apply, Meridian never operates its own Terraform runner
gcloud infra-manager deployments apply meridian-staging-env \
  --location=us-central1 \
  --service-account=projects/meridian-cicd/serviceAccounts/infra-manager@meridian-cicd.iam.gserviceaccount.com \
  --git-source-repo=https://github.com/meridian-logistics/gcp-infra \
  --git-source-directory=environments/staging \
  --git-source-ref=main

Config Connector: Kubernetes-Native GCP Resource Management#

Config Connector takes a structurally different approach from Terraform: instead of a plan/apply CLI workflow, it's a Kubernetes controller (a Custom Resource Definition, or CRD, per GCP resource type) running inside a GKE cluster. You declare a GCP resource as a Kubernetes YAML manifest, and Config Connector's controller continuously reconciles the live GCP resource to match — the exact same reconciliation loop pattern a Deployment or Service object already uses for application workloads.

# A Cloud Storage bucket declared as a Kubernetes resource — applied
# with kubectl (or synced via Config Sync, covered next), NOT with
# terraform apply. Config Connector's controller, running in-cluster,
# does the reconciliation.
apiVersion: storage.cnrm.cloud.google.com/v1beta1
kind: StorageBucket
metadata:
  name: meridian-shipment-exports
  namespace: config-connector
spec:
  location: US
  uniformBucketLevelAccess: true
  lifecycleRule:
    - action:
        type: Delete
      condition:
        age: 90

💡 The transferable insight: Config Connector's real value shows up when application and infrastructure genuinely belong together in one review — a team shipping a new microservice that needs its own dedicated Pub/Sub topic and Cloud Storage bucket can declare all three (the app's Deployment, the topic, the bucket) in the same Kubernetes manifest set, reviewed in the same pull request, applied by the same kubectl apply or GitOps sync. Terraform can absolutely manage the same resources, but it lives in a separate tool, a separate repo convention, and a separate mental model from the Kubernetes-native team already reviewing Deployment YAML all day.

GitOps: The Delivery Model Underneath All of This#

Every tool in this chapter — Terraform, Infrastructure Manager, Config Connector — answers "how do I declare infrastructure." GitOps answers a different question: what triggers a change to actually apply, and where does the record of every change live? The GitOps model's core rule is simple and worth stating precisely: Git is the single source of truth for desired state, and an automated process — never a human running a command from a laptop — reconciles the live system to match whatever Git currently says.

What to notice: the human's only lever is the pull request review — once merged, no person runs the apply command by hand, which is exactly the property that eliminates "it worked when I ran it locally" as a failure mode, because there is no "locally" in this model at all.

This principle applies identically whether the thing being reconciled is a Terraform-managed VPC or a Config-Connector-managed Kubernetes resource — GitOps is a delivery philosophy layered on top of either tool, not a competitor to them. Part 4's Cloud Deploy chapter applies this same principle to application deployments; this chapter is specifically about applying it to infrastructure.

Config Sync: GitOps for What Runs Inside the Cluster#

Config Sync is Google's own managed implementation of the GitOps reconciliation loop, purpose-built for Kubernetes-shaped configuration — it watches a Git repository and continuously applies whatever it finds there to one or many GKE clusters, reverting any manual kubectl drift automatically. It's the mechanism that actually runs the "Reconciling → Applied" transition in the diagram above, when the thing being reconciled is Kubernetes YAML — including Config Connector manifests, since those are just another Kubernetes resource type from Config Sync's point of view.

# A RootSync object — Config Sync's own configuration — pointing a GKE
# cluster at Meridian's platform-config repository. Once applied once
# (by hand, or by the cluster's own bootstrap), every subsequent change
# to that Git repo reaches the cluster with no further manual step.
apiVersion: configsync.gke.io/v1beta1
kind: RootSync
metadata:
  name: root-sync
  namespace: config-management-system
spec:
  sourceFormat: unstructured
  git:
    repo: https://github.com/meridian-logistics/platform-config
    branch: main
    dir: clusters/staging
    auth: gcpserviceaccount
    gcpServiceAccountEmail: config-sync@meridian-staging.iam.gserviceaccount.com

🚨 Incident-critical action: if Config Sync ever reports a persistent SyncError, treat it as equivalent to a failed deploy, not a background warning to triage later — a cluster stuck unable to reconcile means every subsequent Git change to that cluster's config is silently not being applied, and the gap between "what Git says" and "what's actually running" grows with every commit until someone notices, at which point the eventual catch-up sync can apply a much larger, riskier batch of changes at once than any single commit would have been.

Choosing Between Terraform, Config Connector, and Infrastructure Manager#

These three tools solve overlapping problems from different angles, and the exam (and real platform design) expects you to pick deliberately rather than by habit.

💡 A field-tested combination, not a single "winner": most real GCP platforms — Meridian's included — run Terraform (via Infrastructure Manager or self-hosted CI) for foundational, shared infrastructure from Part 1's tooling project, and Config Connector for workload-adjacent resources a feature team owns alongside their own application manifests. This mirrors the earlier web-search finding that these tools "are not competitors and occupy three layers" of the stack — provisioning foundations, workload-coupled resources, and release delivery are each better served by a different one.

A Full Worked Example: Meridian's Environment Factory Module#

Bringing every piece of this chapter together: here's the actual Terraform module Priya's team built to replace the hand-run gcloud bootstrap from Part 1 — a reusable "environment factory" that creates a new environment project with its policy, IAM, and API enablement in one reviewed, reproducible change, run through Infrastructure Manager rather than a self-hosted runner.

# modules/environment/main.tf — one module, instantiated once per
# environment, encoding every Part 1 decision (folder placement,
# environment-specific policy, tooling-project cross-IAM) as code.
variable "environment_name" {
  type = string
}
variable "environment_folder_id" {
  type = string
}
variable "billing_account_id" {
  type = string
}
variable "allow_external_ips" {
  type    = bool
  default = false
}
variable "deployer_service_account_email" {
  type = string
}
variable "deployer_role" {
  type = string
}

resource "google_project" "this" {
  name            = "Meridian ${title(var.environment_name)}"
  project_id      = "meridian-${var.environment_name}"
  folder_id       = var.environment_folder_id
  billing_account = var.billing_account_id
  labels = {
    environment = var.environment_name
    managed_by  = "terraform"
  }
}

resource "google_project_service" "apis" {
  for_each = toset([
    "compute.googleapis.com",
    "container.googleapis.com",
    "run.googleapis.com",
    "secretmanager.googleapis.com",
  ])
  project = google_project.this.project_id
  service = each.value
}

# Environment-specific policy from Part 1's comparison table, now
# encoded once per environment instead of remembered and reapplied
resource "google_project_organization_policy" "external_ip" {
  project    = google_project.this.project_id
  constraint = "constraints/compute.vmExternalIpAccess"
  dynamic "list_policy" {
    for_each = var.allow_external_ips ? [] : [1]
    content {
      deny { all = true }
    }
  }
}

# Cross-project deploy IAM from Part 1, now created alongside the
# project itself instead of as a separate manual follow-up step
resource "google_project_iam_member" "deployer" {
  project = google_project.this.project_id
  role    = var.deployer_role
  member  = "serviceAccount:${var.deployer_service_account_email}"
}

# environments/staging/main.tf — instantiating the module for one
# environment. A fourth environment is now a ~10-line file, not a
# re-run of Part 1's whole gcloud sequence from memory.
module "staging" {
  source                          = "../../modules/environment"
  environment_name                = "staging"
  environment_folder_id           = data.google_active_folder.platform.name
  billing_account_id              = var.billing_account_id
  allow_external_ips              = false
  deployer_service_account_email  = "meridian-deployer@meridian-cicd.iam.gserviceaccount.com"
  deployer_role                   = "roles/run.developer"
}

🧪 Hands-on checkpoint: run terraform plan against this module with allow_external_ips = true for a dev instantiation and false for staging/prod, and confirm the plan output shows the org policy resource differing per environment exactly as Part 1's comparison table specified — a concrete, reviewable confirmation that policy divergence between environments is now enforced by code, not by whoever remembers to run the right gcloud command.

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A team's Terraform configuration for meridian-staging currently uses local state, and two engineers have started alternating who runs apply. What specific failure mode does this setup risk, and what's the minimal fix?

Answer: The risk is exactly this chapter's From-the-Trenches scenario — an engineer running apply from stale local state can produce a plan proposing to recreate resources the other engineer's more recent apply already created or modified, because neither state file has a record of the other's changes. The minimal fix is migrating to a GCS backend (terraform init -migrate-state after adding the backend "gcs" block) — this adds real locking and a single shared source of truth, with no change to the actual resource configuration required.

Problem 2: A platform team wants a new microservice's dedicated Pub/Sub topic and its Kubernetes Deployment reviewed in the exact same pull request, by the same reviewer, using the same tooling the team already uses daily. Which of this chapter's tools fits that requirement best, and why not the alternative?

Answer: Config Connector — declaring the Pub/Sub topic as a PubSubTopic Kubernetes manifest alongside the Deployment YAML lets both live in the same repository, the same PR, reviewed with the same kubectl-native workflow the team already uses for application code. Terraform could technically manage the same topic, but it would live in a separate repository/tool convention, splitting one logical change (a new service and its infra) across two review processes — exactly the friction Config Connector's Kubernetes-native model is designed to remove.

Problem 3: Six months after adopting Config Sync, a team notices a GKE cluster's live configuration has drifted from what's in the platform-config Git repository, but no alert ever fired. What's the most likely root cause, and what should have caught it earlier?

Answer: The most likely root cause is a persistent SyncError that went untriaged — Config Sync stops reconciling on a failure until the underlying issue (invalid YAML, a missing RBAC permission, a broken CRD reference) is fixed, and every commit after that point silently never reaches the cluster. What should have caught it earlier is exactly this chapter's "incident-critical action" callout: alerting on SyncError status the same way a failed deploy would page on-call, rather than treating GitOps reconciliation as a background process nobody actively monitors.

Summary and What's Next#

This chapter built the "how infrastructure is defined and kept in sync" layer underneath Part 1's organizational design: Terraform with remote GCS state as the default declarative tool, Cloud Foundation Toolkit and Fabric FAST as Google's own starting-point blueprints, Infrastructure Manager as a managed alternative to operating your own Terraform runner, Config Connector for workload-coupled resources reviewed alongside application code, and Config Sync as the concrete GitOps reconciliation engine for anything Kubernetes-shaped. Meridian's environment factory module is the tangible result — a fourth environment is now a ten-line instantiation of reviewed, reproducible code instead of a re-run of hand-typed commands.

Part 3 shifts from infrastructure to application delivery: how Cloud Build turns a Git commit into a tested, versioned artifact — the first half of the actual CI/CD pipeline this whole course exists to build.