# GCP DevOps & CI/CD Platform — Part 4: Continuous Delivery with Cloud Deploy

> **Series:** GCP DevOps & CI/CD Platform (4 of 6) — aligned to Professional Cloud DevOps Engineer (PCDE) exam sections 1-2
> **Part 1:** `01-designing-a-devops-organization.md` — Designing a DevOps Organization
> **Part 2:** `02-infrastructure-as-code-and-gitops.md` — Infrastructure as Code & GitOps
> **Part 3:** `03-continuous-integration-with-cloud-build.md` — Continuous Integration with Cloud Build
> **Part 4:** This file — Continuous Delivery with Cloud Deploy
> **Part 5:** `05-managing-environments-and-secrets.md` — Managing Environments, Configuration & Secrets
> **Part 6:** `06-securing-the-deployment-pipeline.md` — Securing the Deployment Pipeline & Dev Environments
> **Questions:** `questions.md`

## Table of Contents

1. [Where Cloud Build Ends and Cloud Deploy Begins](#where-cloud-build-ends-and-cloud-deploy-begins)
2. [Delivery Pipelines and Targets: The Core Objects](#delivery-pipelines-and-targets-the-core-objects)
3. [Skaffold: How Cloud Deploy Actually Renders and Applies Manifests](#skaffold-how-cloud-deploy-actually-renders-and-applies-manifests)
4. [Deployment Strategies: Standard, Canary, and Blue/Green](#deployment-strategies-standard-canary-and-bluegreen)
5. [Approval Gates: Where a Human Stays in the Loop](#approval-gates-where-a-human-stays-in-the-loop)
6. [Automated Promotion, Retry, and Rollback](#automated-promotion-retry-and-rollback)
7. [Auditing and Tracking Every Deployment](#auditing-and-tracking-every-deployment)
8. [A Full Worked Pipeline: Meridian's Production Promotion Path](#a-full-worked-pipeline-meridians-production-promotion-path)
9. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps)
10. [Worked Practice Problems](#worked-practice-problems)
11. [Summary and What's Next](#summary-and-whats-next)

## Where Cloud Build Ends and Cloud Deploy Begins

Part 3 ended with a versioned container image sitting in Artifact Registry — tested, scanned, tagged with its commit SHA. **Cloud Deploy picks up exactly there**: it takes that already-built artifact and manages its promotion through a defined sequence of environments (dev, staging, prod), applying whichever deployment strategy each environment needs, with approval gates and rollback built in as first-class concepts rather than bolted-on scripting.

This is the design boundary the exam guide draws too: section 1.3 ("designing a CI/CD architecture") separates "Continuous integration (CI) with Cloud Build" from "Continuous delivery (CD) with Cloud Deploy" as genuinely distinct concerns — one produces an artifact, the other decides where and how that artifact runs. Confusing the two is a real design smell: a Cloud Build step that deploys directly to production, skipping Cloud Deploy's promotion model entirely, throws away every approval gate and rollback mechanism this chapter covers.

```mermaid
flowchart LR
    subgraph ci["Part 3: Cloud Build (CI)"]
        Source["Source"] --> Test["Test, scan, build"]
        Test --> Artifact["Artifact in<br/>Artifact Registry"]
    end
    subgraph cd["Part 4: Cloud Deploy (CD)"]
        Artifact --> Release["Release created"]
        Release --> DevRO["Rollout: dev"]
        DevRO --> StgRO["Rollout: staging"]
        StgRO --> ProdRO["Rollout: prod<br/>(approval + canary)"]
    end

    classDef ci fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
    classDef cd fill:#f0e9fb,stroke:#6d43c0,color:#10161c
    classDef risky fill:#fbeee0,stroke:#b8650f,color:#10161c
    class Source,Test,Artifact ci
    class Release,DevRO,StgRO cd
    class ProdRO risky
```

**What to notice**: exactly one artifact flows through the whole right-hand side — Cloud Deploy never rebuilds anything, it only ever *promotes* what CI already produced, which is the same "build once, promote the same image" principle Part 1 established for Artifact Registry.

## Delivery Pipelines and Targets: The Core Objects

A **DeliveryPipeline** declares the ordered sequence of environments a release moves through; each environment is a separate **Target**. Both are declared as YAML and applied with `gcloud deploy apply`, in the same declarative, reviewable spirit as Part 2's Terraform and Config Connector resources.

```yaml
# clouddeploy.yaml — Meridian's pipeline definition. Three stages,
# matching the three environment projects from Part 1.
apiVersion: deploy.cloud.google.com/v1
kind: DeliveryPipeline
metadata:
  name: shipment-api-pipeline
  description: Promotion pipeline for shipment-api
serialPipeline:
  stages:
    - targetId: dev
    - targetId: staging
    - targetId: prod
      strategy:
        canary:
          runtimeConfig:
            kubernetes:
              serviceNetworking:
                service: shipment-api
                deployment: shipment-api
          canaryDeployment:
            percentages: [25, 50]
          verify: true

---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: dev
gke:
  cluster: projects/meridian-dev/locations/us-central1/clusters/meridian-dev

---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: staging
gke:
  cluster: projects/meridian-staging/locations/us-central1/clusters/meridian-staging

---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: prod
requireApproval: true
gke:
  cluster: projects/meridian-prod/locations/us-central1/clusters/meridian-prod
```

**What to notice matching Part 1's design**: each `Target` points into a *separate* environment project via its own `gke.cluster` field, exactly the tooling-project-deploys-across-boundaries pattern from Part 1 — this `clouddeploy.yaml` lives in `meridian-cicd`, but its three targets each reach into a different project.

| Object | Declares | Roughly analogous to |
|---|---|---|
| `DeliveryPipeline` | The ordered sequence of environments a release moves through | A CI/CD "stages" list in a Jenkinsfile or GitHub Actions workflow |
| `Target` | One environment's connection details and promotion rules | An environment/deploy-target block in most CD tools |
| `Release` | One immutable, versioned snapshot of what's being promoted | A tagged build artifact plus its render output |
| `Rollout` | One specific attempt to deploy a `Release` to one `Target` | A single deploy execution/run |

## Skaffold: How Cloud Deploy Actually Renders and Applies Manifests

Cloud Deploy doesn't invent its own manifest-templating format — it delegates rendering and applying to **Skaffold**, calling `skaffold render` to turn a `skaffold.yaml` plus Kubernetes manifests (or Helm charts, or Kustomize overlays) into final, environment-specific YAML, then `skaffold apply` to actually apply it to the target cluster.

```yaml
# skaffold.yaml — references Kustomize, letting each environment
# override image tags and replica counts from a shared base
apiVersion: skaffold/v4beta11
kind: Config
manifests:
  kustomize:
    paths:
      - k8s/base
deploy:
  kubectl: {}
profiles:
  - name: prod
    patches:
      - op: replace
        path: /manifests/kustomize/paths/0
        value: k8s/overlays/prod
```

💡 **The transferable insight**: separating *rendering* (turning templates into final YAML) from *deploying* (applying that YAML to a cluster) is the same two-phase design Terraform's `plan`/`apply` split and Helm's `template`/`install` split both use — a rendered-but-not-yet-applied output is inspectable and diffable before anything touches a live cluster, which is exactly the property that makes Cloud Deploy's canary and approval mechanics (next sections) possible to reason about safely.

> [!NOTE]
> Skaffold also supports Helm charts directly as an alternative to Kustomize — pick whichever manifest-templating tool your team already standardized on; Cloud Deploy is agnostic to which one sits underneath Skaffold.

## Deployment Strategies: Standard, Canary, and Blue/Green

A `Target` with no `strategy` block uses Cloud Deploy's default **standard** strategy — the full release replaces the previous one in one step, the simplest and riskiest option, appropriate for `dev` where blast radius is deliberately cheap (Part 1's environment policy table).

**Canary** — shown in this chapter's `clouddeploy.yaml` above — shifts a defined percentage of traffic to the new version, pauses (optionally running a `verify` phase), then shifts more, until 100%. The `percentages: [25, 50]` array means: deploy to 25% of traffic, verify, then 50%, verify, then implicitly complete to 100%.

**Blue/green** keeps the old ("blue") and new ("green") versions both fully deployed simultaneously, cutting traffic over in one atomic step rather than gradually — useful when a gradual traffic split isn't meaningful for the workload (a batch job, a stateful service where partial-traffic makes reasoning about consistency harder) but an instant, easily-reversed cutover still matters.

```mermaid
stateDiagram-v2
    [*] --> Deploying: New release triggers rollout
    Deploying --> Canary25: Canary strategy:<br/>25% of traffic
    Canary25 --> Verifying1: verify: true
    Verifying1 --> Canary50: Verification passed
    Verifying1 --> RolledBack: Verification failed
    Canary50 --> Verifying2: verify: true
    Verifying2 --> Full: Verification passed
    Verifying2 --> RolledBack: Verification failed
    Full --> [*]: 100% on new version
    RolledBack --> [*]: Traffic reverted to<br/>last good release

    classDef progress fill:#e5f0fa,stroke:#1d6fb8,color:#10161c
    classDef verify fill:#f0e9fb,stroke:#6d43c0,color:#10161c
    classDef success fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    classDef failure fill:#fbe8e6,stroke:#b3261e,color:#10161c
    class Deploying,Canary25,Canary50 progress
    class Verifying1,Verifying2 verify
    class Full success
    class RolledBack failure
```

**What to notice**: a failed verification at *either* checkpoint routes to the same `RolledBack` state, not just the final one — canary's whole point is catching a bad release at 25% exposure instead of 100%, and that only works if verification actually gates progression rather than running purely for observability.

| Strategy | Traffic exposure to a bad release | Rollback speed | Best fit |
|---|---|---|---|
| Standard | 100% immediately | Requires a full redeploy of the prior version | Low-risk environments (dev), or workloads where gradual rollout isn't meaningful |
| Canary | Limited to the current percentage step | Fast — only the canary percentage is affected | Production services with real user traffic and measurable SLIs |
| Blue/green | 0% until the atomic cutover, then 100% | Instant — cut traffic back to blue | Stateful/batch workloads, or when an instant, fully-reversible cutover matters more than gradual exposure |

## Approval Gates: Where a Human Stays in the Loop

`requireApproval: true` on a `Target` — set on `prod` in this chapter's pipeline — pauses the rollout after rendering, before any change reaches the cluster, until someone holding `roles/clouddeploy.approver` on that target explicitly approves it. This is the one deliberate manual checkpoint in an otherwise fully automated pipeline, and it exists specifically because Part 1's environment-policy design decided production's blast radius justifies the friction dev and staging don't need.

```bash
# The approval step itself — a real human action, not automatable
# without deliberately removing the safeguard it exists to provide
gcloud deploy rollouts approve shipment-api-prod-rollout-001 \
  --delivery-pipeline=shipment-api-pipeline \
  --release=shipment-api-v42 \
  --target=prod \
  --project=meridian-cicd
```

> [!IMPORTANT]
> `roles/clouddeploy.approver` should be granted to a small, deliberate group — the same "least privilege per target" discipline from Part 1's cross-project IAM design applies here too. Granting it broadly defeats the point of the gate; if everyone can approve everything, the gate is a formality rather than a real second set of eyes.

## Automated Promotion, Retry, and Rollback

Not every promotion step needs a human, and not every failure needs a person paged immediately — Cloud Deploy's **automation rules**, attached to the `DeliveryPipeline`, express this directly as policy rather than external scripting.

```yaml
# Automation rules — attached to the same DeliveryPipeline from
# earlier in this chapter. Auto-promotes dev -> staging after a
# scheduled window, and defines the retry/rollback policy for prod.
apiVersion: deploy.cloud.google.com/v1
kind: Automation
metadata:
  name: shipment-api-automation
  annotations:
    deploy.cloud.google.com/pipeline: shipment-api-pipeline
serviceAccount: automation@meridian-cicd.iam.gserviceaccount.com
selector:
  targets:
    - id: staging
rules:
  - promoteReleaseRule:
      id: auto-promote-to-staging
      wait: 0s
  - repairRolloutRule:
      id: prod-repair
      retry:
        attempts: 2
        backoffMode: BACKOFF_MODE_EXPONENTIAL
      rollback:
        destinationPhase: stable
```

This declares two distinct policies working together: **`promoteReleaseRule`** auto-promotes a release from `dev` to `staging` with no human trigger at all, appropriate because staging's environment policy (Part 1) doesn't require the friction production does. **`repairRolloutRule`** defines what happens when a rollout actually *fails* verification: retry up to 2 times with exponential backoff (catching transient issues — a flaky health check, a momentary resource contention) before giving up and automatically rolling back to the last known-good release.

```mermaid
flowchart TD
    Fail["Rollout verification fails"] --> Retry1{"Retries remaining?"}
    Retry1 -->|"Yes"| Wait["Exponential backoff wait"]
    Wait --> Attempt["Retry the rollout"]
    Attempt --> Fail
    Retry1 -->|"No, exhausted"| Rollback["Automatic rollback to<br/>last successful release"]
    Rollback --> Alert["Alert on-call<br/>(Part 5's SRE course covers this)"]

    classDef fail fill:#fbe8e6,stroke:#b3261e,color:#10161c
    classDef retry fill:#fbeee0,stroke:#b8650f,color:#10161c
    classDef recover fill:#e5f5ea,stroke:#1f8a4c,color:#10161c
    class Fail fail
    class Retry1,Wait,Attempt retry
    class Rollback,Alert recover
```

🔍 **From the Trenches**: A team configured automated rollback with zero retries — any single verification failure triggered an immediate rollback. A transient DNS resolution blip inside the cluster, lasting under two seconds, caused a health check to fail exactly once during a canary's 25% step, triggering a full automatic rollback of an otherwise completely healthy release. Engineers spent forty minutes investigating "why did the deploy fail" before finding the real answer buried in cluster DNS logs: nothing about the release was wrong at all. The two-levels-deep lesson: the surface symptom was a confusing rollback with no code-level cause, the immediate cause was zero retry tolerance for a genuinely transient failure class, and the underlying condition was that the automation policy had been configured to treat *every* verification failure as equally meaningful, with no distinction between "the new code is actually broken" and "something outside the release flaked once." Adding `attempts: 2` with exponential backoff, as shown in the YAML above, fixed it without weakening the safety the rollback rule exists to provide.

## Auditing and Tracking Every Deployment

The exam guide's "auditing and tracking deployments" consideration (2.2) has a direct, concrete answer with Cloud Deploy: every `Release` and `Rollout` is itself a durable, queryable Cloud Deploy resource — `gcloud deploy releases list` and `gcloud deploy rollouts list` give a complete history with no separate logging system required, and every action (a promotion, an approval, a rollback) is captured in Cloud Audit Logs automatically, attributed to the identity that triggered it.

```bash
# "What's actually running in prod, and who approved it" — answerable
# directly, no separate deployment-tracking spreadsheet required
gcloud deploy rollouts list \
  --delivery-pipeline=shipment-api-pipeline \
  --target=prod \
  --project=meridian-cicd \
  --format="table(name, state, createTime)"
```

> [!TIP]
> **Best Practice**: don't build a separate deployment-tracking system alongside Cloud Deploy "just to be safe" — the `Release`/`Rollout` history plus Cloud Audit Logs already answers "what's running, since when, promoted by whom" with no additional infrastructure. A parallel tracking system is one more thing to keep in sync and one more place the truth can drift from what Cloud Deploy itself already knows authoritatively.

## A Full Worked Pipeline: Meridian's Production Promotion Path

Putting every piece of this chapter together — the actual sequence from a merged commit to a fully promoted production release, incorporating the automation and canary configuration from earlier sections.

```bash
# 1. Cloud Build (Part 3) finishes, pushes the image, and creates the
#    Cloud Deploy release referencing it
gcloud deploy releases create shipment-api-v42 \
  --delivery-pipeline=shipment-api-pipeline \
  --project=meridian-cicd \
  --region=us-central1 \
  --images=shipment-api=us-central1-docker.pkg.dev/meridian-cicd/meridian-images/shipment-api:a1b2c3d

# 2. Cloud Deploy automatically creates and runs the dev rollout first
#    (the pipeline's first stage, no approval or canary required)

# 3. The automation rule auto-promotes dev -> staging with no human
#    trigger, per this chapter's Automation resource

# 4. Staging's rollout runs; if it passes, prod's rollout is created
#    but PAUSED, because prod has requireApproval: true

# 5. A human with roles/clouddeploy.approver reviews and approves
gcloud deploy rollouts approve shipment-api-v42-to-prod-0001 \
  --delivery-pipeline=shipment-api-pipeline \
  --release=shipment-api-v42 \
  --target=prod \
  --project=meridian-cicd

# 6. Cloud Deploy runs the canary strategy: 25% traffic, verify,
#    50% traffic, verify, then completes to 100% -- with automatic
#    retry (2 attempts) and rollback-on-exhaustion if verification
#    ever fails, per the Automation resource's repairRolloutRule
```

🧪 **Hands-on checkpoint**: after step 6 completes, run the audit query from the previous section and confirm the `Rollout` history shows the full sequence — dev, staging (auto-promoted), and prod (approval-gated, canary-staged) — as one continuous, queryable record with no gaps and no separate system needed to reconstruct what happened.

## Common Mistakes and Interview Traps

| Mistake | Why it's wrong | What to say/do instead |
|---|---|---|
| Deploying directly from a Cloud Build step, skipping Cloud Deploy | Throws away approval gates, canary strategy, and built-in rollout history | Build in Cloud Build, hand off to Cloud Deploy for every promotion |
| Zero retry tolerance on automated rollback rules | A single transient failure triggers a full rollback of a genuinely healthy release | Configure a small number of retries with backoff before rollback (this chapter's From-the-Trenches fix) |
| Granting `roles/clouddeploy.approver` broadly | Defeats the purpose of a deliberate human checkpoint on production | Grant it to a small, specific group per target |
| Using the standard (all-at-once) strategy for production traffic | 100% of users are exposed to any regression instantly | Use canary for anything with real user traffic and measurable SLIs |
| Building a separate deployment-tracking spreadsheet or dashboard | Duplicates what `Release`/`Rollout` objects and Cloud Audit Logs already track authoritatively | Query Cloud Deploy's own resources and audit logs directly |
| Assuming canary and blue/green solve the same problem | They differ in traffic-exposure shape and rollback mechanics, not just terminology | Canary for gradual, measurable exposure; blue/green for an atomic, fully-reversible cutover |

## Worked Practice Problems

**Problem 1**: A production `Target` currently uses the standard deployment strategy with no approval gate. A recent bad release caused a full-traffic outage for six minutes before anyone noticed. What two independent changes would most directly address this, and why are both worth making rather than just one?

*Answer*: Add `requireApproval: true` (a human reviews before any change reaches the cluster) and switch to a canary strategy with a `verify` phase (limits exposure to a fraction of traffic and catches a regression automatically before it reaches 100%). They're independent and complementary: approval catches problems a human can recognize before deployment starts, while canary+verify catches problems that only manifest once real traffic hits the new version — a human approving a release doesn't guarantee the release is actually healthy under load, and a canary alone doesn't stop a change nobody should have approved in the first place from starting its rollout.

**Problem 2**: An `Automation` resource has a `repairRolloutRule` with `attempts: 2` but no `rollback` block defined. What happens if both retries fail, and is this a safe default to leave as-is for a production target?

*Answer*: Per Cloud Deploy's automation behavior, if no rollback is configured (or configured retries are exhausted with no rollback rule), a new rollout is created to roll back to the most recently successful release on that target — rollback still happens, it's just Cloud Deploy's own default fallback behavior. Whether this is "safe to leave as-is" depends on whether that default fallback's exact behavior (target phase, verification requirements on the rollback rollout itself) matches what the team actually wants; the safer, more explicit choice is still to declare the `rollback` block deliberately rather than depend on undocumented default behavior nobody on the team consciously chose.

**Problem 3**: Meridian wants `dev` to auto-promote from every merged commit with zero manual steps, but wants `staging` to require someone to explicitly kick off promotion (without needing full production-grade approval). How would you configure this across `Target` and `Automation` resources?

*Answer*: Leave `dev`'s `Target` with no `requireApproval` and give it a `promoteReleaseRule` in an `Automation` resource so every dev rollout completion automatically creates the next release for promotion. For `staging`, don't add a `promoteReleaseRule` selector targeting it (so nothing auto-promotes into it) and also don't set `requireApproval: true` on its `Target` (since that specifically gates *deployment* to that target with the formal approver-role mechanism) — instead, promotion into staging happens via an explicit, manually-run `gcloud deploy releases promote --to-target=staging` command, giving a human a deliberate trigger without the heavier formal-approval audit trail reserved for prod.

## Summary and What's Next

This chapter completed the CI/CD picture Part 3 started: Cloud Deploy's `DeliveryPipeline`/`Target`/`Release`/`Rollout` object model, Skaffold as the rendering engine underneath every promotion, the standard/canary/blue-green strategy tradeoffs, approval gates as the one deliberate human checkpoint, and automation rules that turn promotion, retry, and rollback into declared policy rather than external scripting. Meridian's production promotion path — auto-promoted staging, approval-gated and canary-staged production, with retry-then-rollback built in — is the concrete pipeline every prior chapter's design decisions (the tooling project, the environment policy differences, the fail-fast CI ordering) were ultimately building toward.

**Part 5** turns to what flows *through* this pipeline besides the container image itself: environment-specific configuration and secrets — how Meridian keeps `dev`, `staging`, and `prod` configuration correctly separated without hardcoding a single value into the pipeline definition itself.