Designing a DevOps Organization

Assumes you're comfortable with GCP's resource hierarchy, IAM, and service accounts at the level covered in GCP Cloud Engineer Foundations — this series builds the DevOps-specific layer on top of that base rather than re-teaching it.

Table of Contents#

1. Why This Course Exists — Professional Cloud DevOps Engineer
2. Meridian Logistics: The Platform Team Scales Up
3. What Makes PCDE Different From ACE
4. Designing a Resource Hierarchy for CI/CD
5. The Shared Tooling Project Pattern
6. Environment-Specific Policy Differences
7. Data Residency for Pipelines and Artifacts
8. Google-Recommended Practices and Blueprints
9. Automation With Scripting: Python and Go on GCP
10. Terminology Map: DevOps Tooling Across AWS, Azure, and GCP
11. A Full Worked Bootstrap: Meridian's CI/CD Landing Zone
12. Common Mistakes and Interview Traps
13. Worked Practice Problems
14. Summary and What's Next

Why This Course Exists — Professional Cloud DevOps Engineer#

This course teaches everything Google's Professional Cloud DevOps Engineer (PCDE) exam tests under "bootstrapping an organization" and "building CI/CD pipelines" — roughly 45% of the exam by weight. The PCDE credential certifies that you can implement the processes and tooling that let a team ship software and infrastructure changes quickly without sacrificing reliability — the practical intersection of software engineering discipline and operations ownership that the industry calls DevOps or platform engineering, depending on which decade coined the job title at your employer.

Google's own exam guide splits the certification into five sections: bootstrapping an organization (20%), building CI/CD pipelines (25%), applying SRE practices (18%), observability and troubleshooting (25%), and performance/cost optimization (12%). This course covers the first two sections in full depth — the mechanical foundation of designing an org for continuous delivery and building the pipelines themselves. GCP SRE & Observability, the next course in this series, picks up the remaining three sections.

Meridian Logistics: The Platform Team Scales Up#

If you worked through GCP Cloud Engineer Foundations, you already know Meridian Logistics — a mid-size freight-tracking company migrating from a rented data center into GCP. Its platform team of three (Priya on hierarchy/IAM, Devon on compute/networking, Ana on data/observability) got the foundational landing zone bootstrapped in that course.

This course picks up six months later. Meridian now runs four services in production on GCP — shipment-api, an orders Cloud SQL database, a Pub/Sub-backed GPS ingestion pipeline, and a BigQuery analytics warehouse — and the team has grown to eight engineers split across two feature squads. The problem that got Priya promoted to Platform Engineering Lead: every squad was hand-running gcloud commands from their laptops to deploy, nobody could say with confidence what was actually running in production versus what was in a main branch nobody had deployed yet, and a Friday-afternoon deploy of shipment-api had just taken down GPS ingestion for eleven minutes because the two services shared a Cloud SQL instance and nobody had a staging environment to catch the connection-pool exhaustion before it hit customers.

That incident is the reason this course exists inside the story: Meridian needs a real CI/CD platform, not tribal knowledge and lucky timing. Every chapter in this series builds one deliberate piece of what Priya's team builds in response.

What to notice: the incident that triggers this course is a governance failure, not a technology gap — Meridian already had GCP, Cloud SQL, and Pub/Sub. What was missing was the organizational structure and pipeline discipline to use them safely at more than one team's scale.

What Makes PCDE Different From ACE#

The Associate Cloud Engineer exam certifies that you can operate GCP's building blocks correctly. PCDE certifies something structurally different: that you can design the system those building blocks get deployed through, and that the system stays reliable as change velocity increases. A useful way to state the shift in one sentence: ACE asks "can you deploy this VM correctly," PCDE asks "can you build the pipeline that deploys hundreds of VMs correctly, every day, without a human approving each one."

💡 The transferable insight: this same ACE-to-DevOps shift exists on every cloud, not just GCP — AWS's SysOps/Solutions Architect Associate versus its DevOps Engineer Professional draws almost the identical line, and Azure Administrator versus Azure DevOps Engineer Expert does too. If you've already made this jump on another cloud, the concepts transfer; only the product names change.

Designing a Resource Hierarchy for CI/CD#

The Foundations course covered the resource hierarchy — organization, folders, projects — as GCP's isolation and policy-inheritance mechanism. For a DevOps organization specifically, one more design question sits on top of that: which projects hold environments, and which project holds the pipeline machinery that deploys into them?

Two patterns cover almost every real organization, and Google's own guidance leans toward the second one as the codebase and team count grow past a handful of services.

Pattern 1 — per-environment projects, pipeline runs from inside each one. Every environment (dev, staging, prod) is its own project. The CI/CD pipeline itself — the Cloud Build triggers, the Artifact Registry repository — lives duplicated inside each environment's project. This is simple to reason about early on: one team, one service, no shared blast radius. It falls apart once a second team shows up, because now two teams are each maintaining their own copy of the same pipeline logic, and a security fix to the pipeline has to be applied N times.

Pattern 2 — a dedicated tooling project, environments stay separate, the pipeline deploys across the boundary. One project (commonly named something like meridian-cicd or meridian-tooling) holds Cloud Build, Artifact Registry, and Cloud Deploy. It has no application workloads. Cross-project IAM bindings grant that project's service accounts the specific, narrow permissions needed to deploy into meridian-dev, meridian-staging, and meridian-prod. This is the pattern PCDE expects you to reach for by default, because it centralizes pipeline security review to one place and lets every team's pipeline share the same audited, hardened base configuration.

What to notice: the arrow into meridian-prod is drawn differently on purpose — it's the one deploy target that needs a human approval gate and the narrowest service-account scope, which Part 4 covers when it builds Cloud Deploy's approval-required promotion.

🔍 From the Trenches: A team migrating from a single shared "everything" project to per-environment projects skipped the tooling-project step and instead granted their Cloud Build service account roles/editor on all three environment projects "temporarily, to get the migration done faster." The migration took four months instead of the planned three weeks, because nobody wanted to be the one to revoke the broad grant once things were finally working, and revoking it required rewriting every build step that had quietly started depending on permissions it never should have had — enabling APIs mid-build, reading Secret Manager entries the pipeline author never explicitly requested. The two-levels-deep lesson: the surface symptom was "the migration is taking too long," the immediate cause was scope creep in a temporary IAM grant, and the underlying condition was that nobody had defined what the pipeline's actual required permission set was before granting it broad access — so there was no baseline to audit against once "temporary" quietly became permanent.

The Shared Tooling Project Pattern#

Once a tooling project exists, three more decisions follow directly from it.

Artifact Registry lives in the tooling project, not per-environment. A container image built once from a given commit should be the exact same image promoted through dev, staging, and prod — rebuilding per environment (even from identical source) breaks the supply-chain guarantee that what was tested is what ships, and doubles build cost for no benefit. One Artifact Registry repository per artifact type (a Docker repo, a Maven repo) in the tooling project, referenced by every environment's deploy target.

Cloud Build triggers live in the tooling project, and reference source in a separate Git repository (or repositories) that itself does not live inside any environment project. GCP's Cloud Source Repositories or an external Git host (GitHub, GitLab) both work; the exam guide explicitly calls out "widely used third-party tooling (e.g., Git, Jenkins, Argo CD, Packer, kpt)" as fair game — PCDE is not GCP-tool-exclusive.

Cross-project IAM is the connective tissue, and it should be least-privilege per target. A service account in the tooling project (say, deployer@meridian-cicd.iam.gserviceaccount.com) gets roles/run.developer in meridian-dev, a similar narrow role in meridian-staging, and in meridian-prod a role plus an approval-gate requirement rather than direct deploy rights.

What Actually Happens During a Cross-Project Deploy#

The IAM binding on meridian-prod grants a permission, but it's worth seeing the actual mechanism underneath "the pipeline deploys across a project boundary" — this is the kind of black-box internal flow most tutorials skip past.

What to notice: at no point does a long-lived credential cross the project boundary — the service account's identity lives in meridian-cicd, and what actually crosses into meridian-prod is a short-lived token plus an API call, checked against that project's own IAM policy on every single request. This is the same token-exchange mechanism Part 6 revisits for Workload Identity Federation with external CI systems — cross-project deploy and cross-cloud federation are the same underlying pattern at different scope.

Environment-Specific Policy Differences#

A resource hierarchy that treats meridian-dev, meridian-staging, and meridian-prod as identically-policed siblings misses the actual point of separating them. Each environment should carry deliberately different org policy constraints, matched to what that environment is for — dev needs to move fast and tolerate experimentation, prod needs to fail closed.

💡 The transferable insight: this is the same "blast-radius-scaled trust" principle behind staged canary rollouts in Part 4, applied one layer up at the environment/policy level instead of the traffic-percentage level — dev is where you accept risk to move fast, prod is where the organization has decided the cost of a mistake outweighs the cost of friction.

Data Residency for Pipelines and Artifacts#

The exam guide explicitly calls out data residency as a resource-hierarchy consideration, and it's easy to overlook for CI/CD specifically because "pipeline data" doesn't feel like customer data. It is, in three concrete ways that matter for a real production platform:

• Build logs and source snapshots — Cloud Build stores build logs and can persist source archives; if source contains customer PII (test fixtures copied from production, for instance — itself a bad practice worth flagging separately), those logs inherit the same residency obligations as the data itself.
• Container images — an image baked with a customer's regional data embedded (a compiled ML model trained on EU customer behavior, say) needs Artifact Registry located in a compliant region, and needs its replication policy checked rather than assumed.
• Secret Manager and Cloud KMS key locations — a secret or a customer-managed encryption key created without an explicit location constraint defaults to a multi-region setting that may not satisfy a data-residency requirement a customer contract actually promises.

Meridian's own answer, once a European logistics customer signed a contract requiring EU data residency: not a resource-hierarchy redesign, but a resource-location org policy (constraints/gcp.resourceLocations) applied at the folder level for that customer's dedicated project, restricting new resources to europe-west1/europe-west4, and a matching Artifact Registry location and Cloud KMS keyring location for anything that project's pipeline touches.

Google-Recommended Practices and Blueprints#

Google publishes opinionated, field-tested starting configurations rather than leaving every organization to invent its landing zone from a blank page — the same instinct behind AWS Landing Zone or Azure Landing Zones, expressed with GCP's own tooling.

Cloud Foundation Toolkit (CFT) ships as a set of Terraform modules (terraform-google-modules on GitHub) implementing Google's own reference architecture for projects, networking, IAM, and logging — the resource hierarchy from the Foundations course, expressed as reusable, versioned Terraform rather than hand-run gcloud commands. Fabric FAST (Fast Adoption of Secure Terraform) is a more prescriptive, opinionated evolution of the same idea from Google's own Professional Services organization, aimed at getting an enterprise-scale multi-environment foundation live quickly with security defaults already applied. Both are starting points to fork and adapt, not black boxes to depend on unmodified — Part 2 goes hands-on with the Terraform layer underneath both.

Automation With Scripting: Python and Go on GCP#

The exam guide lists "automation with scripting (e.g., Python, Go)" as a resource under "managing infrastructure" — a reminder that not every operational task fits cleanly into Terraform or a gcloud one-liner. Real platform teams write small, purpose-built scripts and services for tasks that need actual logic: reconciling drift between two systems, running a scheduled cleanup, or reacting to an event.

Python is the more common choice for one-off operational scripts and Cloud Functions, largely because google-cloud-python client libraries are mature and the barrier to a junior engineer contributing is low. Go is the more common choice for anything that ships as a long-running service or a CLI tool distributed to other engineers, because it compiles to a single static binary with no runtime dependency to manage — genuinely relevant when the "customer" of your automation is another team's laptop, not just your own CI runner.

# A small, real example of the kind of automation PCDE expects you to
# recognize as legitimate infrastructure work, not "just a script":
# reconciling GKE node pools against an expected count, alerting if drift
# is found. Runs as a scheduled Cloud Function, not a cron job on a VM
# nobody patches.
from google.cloud import container_v1

def check_node_pool_drift(project_id: str, location: str, cluster_id: str,
                            expected_pools: dict[str, int]) -> list[str]:
    """Compare live GKE node pool counts against the expected baseline.

    Returns a list of human-readable drift descriptions, empty if none.
    """
    client = container_v1.ClusterManagerClient()
    cluster_name = f"projects/{project_id}/locations/{location}/clusters/{cluster_id}"
    cluster = client.get_cluster(name=cluster_name)

    drift = []
    live_pools = {pool.name: pool.initial_node_count for pool in cluster.node_pools}
    for name, expected_count in expected_pools.items():
        actual = live_pools.get(name)
        if actual is None:
            drift.append(f"Expected pool '{name}' not found on cluster")
        elif actual != expected_count:
            drift.append(f"Pool '{name}' has {actual} nodes, expected {expected_count}")
    return drift

⚙️ The mechanism worth internalizing: this function does not touch Terraform state or a gcloud config file — it calls the same underlying Cloud API that both of those tools call, directly, from application code. PCDE treats this as a first-class DevOps skill precisely because Terraform declares desired state well but doesn't natively alert on drift the way a small scheduled function can; the tools are complementary, not competing.

The Go equivalent shows up more often as a small internal CLI than as a scheduled function — Meridian's own merictl tool wraps a handful of these API calls behind subcommands the whole platform team runs from their laptops, distributed as a single binary through the tooling project's own Artifact Registry (a generic repository, not just Docker):

// merictl drift check — same underlying Container API as the Python
// example above, packaged as a subcommand of an internal CLI instead
// of a scheduled function. Distributed as one static binary so every
// engineer's laptop runs it with zero dependency setup.
package main

import (
	"context"
	"fmt"

	container "cloud.google.com/go/container/apiv1"
	containerpb "cloud.google.com/go/container/apiv1/containerpb"
)

func checkNodePoolDrift(ctx context.Context, projectID, location, clusterID string,
	expected map[string]int32) ([]string, error) {
	client, err := container.NewClusterManagerClient(ctx)
	if err != nil {
		return nil, fmt.Errorf("creating cluster manager client: %w", err)
	}
	defer client.Close()

	name := fmt.Sprintf("projects/%s/locations/%s/clusters/%s", projectID, location, clusterID)
	cluster, err := client.GetCluster(ctx, &containerpb.GetClusterRequest{Name: name})
	if err != nil {
		return nil, fmt.Errorf("fetching cluster: %w", err)
	}

	live := make(map[string]int32)
	for _, pool := range cluster.GetNodePools() {
		live[pool.GetName()] = pool.GetInitialNodeCount()
	}

	var drift []string
	for name, want := range expected {
		got, ok := live[name]
		switch {
		case !ok:
			drift = append(drift, fmt.Sprintf("expected pool %q not found", name))
		case got != want:
			drift = append(drift, fmt.Sprintf("pool %q has %d nodes, expected %d", name, got, want))
		}
	}
	return drift, nil
}

Terminology Map: DevOps Tooling Across AWS, Azure, and GCP#

If you've built CI/CD on another cloud, this table heads off the most common false-equivalence mistake before it costs you time in the exam or in production.

A Full Worked Bootstrap: Meridian's CI/CD Landing Zone#

Bringing this chapter's pieces together, here is the actual sequence Priya's team ran to stand up the tooling project — the concrete artifact this chapter has been building toward.

# 1. Create the dedicated tooling project under the existing
#    "platform" folder from the Foundations course's resource hierarchy
gcloud projects create meridian-cicd \
  --folder=$(gcloud resource-manager folders list \
    --organization=$MERIDIAN_ORG_ID \
    --filter="displayName=platform" --format="value(name)") \
  --name="Meridian CI/CD Tooling"

# 2. Link billing and enable the APIs this project actually needs —
#    nothing more, per the least-privilege habit from Foundations Part 1
gcloud billing projects link meridian-cicd --billing-account=$BILLING_ACCOUNT_ID
gcloud services enable \
  cloudbuild.googleapis.com \
  artifactregistry.googleapis.com \
  clouddeploy.googleapis.com \
  secretmanager.googleapis.com \
  --project=meridian-cicd

# 3. Create the deploy service account that will hold cross-project
#    permissions into each environment — created here, granted there
gcloud iam service-accounts create meridian-deployer \
  --project=meridian-cicd \
  --display-name="Cross-environment Cloud Deploy service account"

# 4. Grant that service account a narrow, target-specific role in EACH
#    environment project individually — never a folder-level grant here,
#    since prod and dev should never share the exact same permission set
gcloud projects add-iam-policy-binding meridian-dev \
  --member="serviceAccount:meridian-deployer@meridian-cicd.iam.gserviceaccount.com" \
  --role="roles/run.developer"

gcloud projects add-iam-policy-binding meridian-prod \
  --member="serviceAccount:meridian-deployer@meridian-cicd.iam.gserviceaccount.com" \
  --role="roles/clouddeploy.jobRunner"

# 5. Create the shared Artifact Registry repository every environment
#    will pull the SAME built image from
gcloud artifacts repositories create meridian-images \
  --project=meridian-cicd \
  --repository-format=docker \
  --location=us-central1 \
  --description="Shared container images for all Meridian services"

🧪 Hands-on checkpoint: run gcloud projects get-iam-policy meridian-prod after step 4 and confirm the deployer service account's binding shows only roles/clouddeploy.jobRunner — not roles/editor, not roles/owner. If your own practice environment shows a broader role, that's the exact scope-creep failure mode from this chapter's "From the Trenches" callout, caught before it becomes a habit.

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: Meridian's meridian-dev project currently has Cloud Build triggers, Artifact Registry, and application workloads all living together in the same project — a pattern left over from before this chapter's redesign. A second team is about to onboard a new service. What's the single highest-priority structural change to make before that onboarding, and why?

Answer: Extract Cloud Build and Artifact Registry into a dedicated tooling project before the second team onboards. If the second team's pipeline is built inside the shared meridian-dev project too, both teams inherit each other's pipeline blast radius (a bad build trigger, a compromised build step) and duplicate maintenance of pipeline configuration begins immediately — exactly the failure mode that caused the Friday incident that opened this chapter. Doing this before the second onboarding is materially cheaper than migrating two teams' pipelines later.

Problem 2: A compliance requirement states that a specific customer's data must never leave europe-west4. The customer's application already runs in a dedicated meridian-eu-prod project with a resource-location org policy constraining new resources to that region. A build for this service still runs through the shared meridian-cicd tooling project in us-central1. Is this a compliance gap, and if so, what specifically needs to change?

Answer: Yes — build logs and any source snapshot Cloud Build retains are themselves data subject to the same residency requirement, and a build running in us-central1 creates and stores that data outside europe-west4 regardless of where the final deployed application lives. The fix is a dedicated regional build path for this customer: either a separate tooling project location-constrained to the EU region, or (if Cloud Build supports a regional endpoint for the required region) explicitly pinning that customer's Cloud Build triggers and any Secret Manager/KMS resources they touch to the compliant region rather than relying on the multi-region default.

Problem 3: Devon proposes granting the meridian-deployer service account roles/owner on meridian-staging "so we stop having to add new IAM bindings every time the pipeline needs one more permission." What should Priya's response be, and what alternative addresses Devon's actual underlying complaint?

Answer: Priya should decline the broad grant — roles/owner on staging gives the pipeline (and anyone who can trigger a build) the ability to change IAM policy, delete resources, and access any data in that project, none of which a deploy pipeline legitimately needs. The underlying complaint (repeatedly adding narrow IAM bindings is friction) is real and worth solving differently: define the deploy service account's required permission set as a custom role or a small, explicit list of predefined roles once, then grant that same defined set to every new environment as it's created — friction goes away without trading away least-privilege.

Summary and What's Next#

This chapter established the organizational shape a real CI/CD platform needs before any pipeline gets built: a dedicated tooling project separate from environment projects, cross-project IAM scoped per target, data-residency awareness that extends to build artifacts and not just application data, and Google's own blueprint tooling (CFT, Fabric FAST) as a starting point rather than a finished answer. Meridian's tooling project — meridian-cicd, with its Artifact Registry repository and deploy service account already scoped correctly — is the concrete foundation every remaining chapter in this course builds on.

Part 2 goes one layer deeper into that foundation: how the infrastructure inside each of these projects gets defined, versioned, and kept in sync with what's actually running — Terraform, Config Connector, Infrastructure Manager, and the GitOps model that ties them together.