Continuous Delivery with Cloud Deploy

Table of Contents#

1. Where Cloud Build Ends and Cloud Deploy Begins
2. Delivery Pipelines and Targets: The Core Objects
3. Skaffold: How Cloud Deploy Actually Renders and Applies Manifests
4. Deployment Strategies: Standard, Canary, and Blue/Green
5. Approval Gates: Where a Human Stays in the Loop
6. Automated Promotion, Retry, and Rollback
7. Auditing and Tracking Every Deployment
8. A Full Worked Pipeline: Meridian's Production Promotion Path
9. Common Mistakes and Interview Traps
10. Worked Practice Problems
11. Summary and What's Next

Where Cloud Build Ends and Cloud Deploy Begins#

Part 3 ended with a versioned container image sitting in Artifact Registry — tested, scanned, tagged with its commit SHA. Cloud Deploy picks up exactly there: it takes that already-built artifact and manages its promotion through a defined sequence of environments (dev, staging, prod), applying whichever deployment strategy each environment needs, with approval gates and rollback built in as first-class concepts rather than bolted-on scripting.

This is the design boundary the exam guide draws too: section 1.3 ("designing a CI/CD architecture") separates "Continuous integration (CI) with Cloud Build" from "Continuous delivery (CD) with Cloud Deploy" as genuinely distinct concerns — one produces an artifact, the other decides where and how that artifact runs. Confusing the two is a real design smell: a Cloud Build step that deploys directly to production, skipping Cloud Deploy's promotion model entirely, throws away every approval gate and rollback mechanism this chapter covers.

What to notice: exactly one artifact flows through the whole right-hand side — Cloud Deploy never rebuilds anything, it only ever promotes what CI already produced, which is the same "build once, promote the same image" principle Part 1 established for Artifact Registry.

Delivery Pipelines and Targets: The Core Objects#

A DeliveryPipeline declares the ordered sequence of environments a release moves through; each environment is a separate Target. Both are declared as YAML and applied with gcloud deploy apply, in the same declarative, reviewable spirit as Part 2's Terraform and Config Connector resources.

# clouddeploy.yaml — Meridian's pipeline definition. Three stages,
# matching the three environment projects from Part 1.
apiVersion: deploy.cloud.google.com/v1
kind: DeliveryPipeline
metadata:
  name: shipment-api-pipeline
  description: Promotion pipeline for shipment-api
serialPipeline:
  stages:
    - targetId: dev
    - targetId: staging
    - targetId: prod
      strategy:
        canary:
          runtimeConfig:
            kubernetes:
              serviceNetworking:
                service: shipment-api
                deployment: shipment-api
          canaryDeployment:
            percentages: [25, 50]
          verify: true

---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: dev
gke:
  cluster: projects/meridian-dev/locations/us-central1/clusters/meridian-dev

---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: staging
gke:
  cluster: projects/meridian-staging/locations/us-central1/clusters/meridian-staging

---
apiVersion: deploy.cloud.google.com/v1
kind: Target
metadata:
  name: prod
requireApproval: true
gke:
  cluster: projects/meridian-prod/locations/us-central1/clusters/meridian-prod

What to notice matching Part 1's design: each Target points into a separate environment project via its own gke.cluster field, exactly the tooling-project-deploys-across-boundaries pattern from Part 1 — this clouddeploy.yaml lives in meridian-cicd, but its three targets each reach into a different project.

Skaffold: How Cloud Deploy Actually Renders and Applies Manifests#

Cloud Deploy doesn't invent its own manifest-templating format — it delegates rendering and applying to Skaffold, calling skaffold render to turn a skaffold.yaml plus Kubernetes manifests (or Helm charts, or Kustomize overlays) into final, environment-specific YAML, then skaffold apply to actually apply it to the target cluster.

# skaffold.yaml — references Kustomize, letting each environment
# override image tags and replica counts from a shared base
apiVersion: skaffold/v4beta11
kind: Config
manifests:
  kustomize:
    paths:
      - k8s/base
deploy:
  kubectl: {}
profiles:
  - name: prod
    patches:
      - op: replace
        path: /manifests/kustomize/paths/0
        value: k8s/overlays/prod

💡 The transferable insight: separating rendering (turning templates into final YAML) from deploying (applying that YAML to a cluster) is the same two-phase design Terraform's plan/apply split and Helm's template/install split both use — a rendered-but-not-yet-applied output is inspectable and diffable before anything touches a live cluster, which is exactly the property that makes Cloud Deploy's canary and approval mechanics (next sections) possible to reason about safely.

Deployment Strategies: Standard, Canary, and Blue/Green#

A Target with no strategy block uses Cloud Deploy's default standard strategy — the full release replaces the previous one in one step, the simplest and riskiest option, appropriate for dev where blast radius is deliberately cheap (Part 1's environment policy table).

Canary — shown in this chapter's clouddeploy.yaml above — shifts a defined percentage of traffic to the new version, pauses (optionally running a verify phase), then shifts more, until 100%. The percentages: [25, 50] array means: deploy to 25% of traffic, verify, then 50%, verify, then implicitly complete to 100%.

Blue/green keeps the old ("blue") and new ("green") versions both fully deployed simultaneously, cutting traffic over in one atomic step rather than gradually — useful when a gradual traffic split isn't meaningful for the workload (a batch job, a stateful service where partial-traffic makes reasoning about consistency harder) but an instant, easily-reversed cutover still matters.

What to notice: a failed verification at either checkpoint routes to the same RolledBack state, not just the final one — canary's whole point is catching a bad release at 25% exposure instead of 100%, and that only works if verification actually gates progression rather than running purely for observability.

Approval Gates: Where a Human Stays in the Loop#

requireApproval: true on a Target — set on prod in this chapter's pipeline — pauses the rollout after rendering, before any change reaches the cluster, until someone holding roles/clouddeploy.approver on that target explicitly approves it. This is the one deliberate manual checkpoint in an otherwise fully automated pipeline, and it exists specifically because Part 1's environment-policy design decided production's blast radius justifies the friction dev and staging don't need.

# The approval step itself — a real human action, not automatable
# without deliberately removing the safeguard it exists to provide
gcloud deploy rollouts approve shipment-api-prod-rollout-001 \
  --delivery-pipeline=shipment-api-pipeline \
  --release=shipment-api-v42 \
  --target=prod \
  --project=meridian-cicd

Automated Promotion, Retry, and Rollback#

Not every promotion step needs a human, and not every failure needs a person paged immediately — Cloud Deploy's automation rules, attached to the DeliveryPipeline, express this directly as policy rather than external scripting.

# Automation rules — attached to the same DeliveryPipeline from
# earlier in this chapter. Auto-promotes dev -> staging after a
# scheduled window, and defines the retry/rollback policy for prod.
apiVersion: deploy.cloud.google.com/v1
kind: Automation
metadata:
  name: shipment-api-automation
  annotations:
    deploy.cloud.google.com/pipeline: shipment-api-pipeline
serviceAccount: automation@meridian-cicd.iam.gserviceaccount.com
selector:
  targets:
    - id: staging
rules:
  - promoteReleaseRule:
      id: auto-promote-to-staging
      wait: 0s
  - repairRolloutRule:
      id: prod-repair
      retry:
        attempts: 2
        backoffMode: BACKOFF_MODE_EXPONENTIAL
      rollback:
        destinationPhase: stable

This declares two distinct policies working together: promoteReleaseRule auto-promotes a release from dev to staging with no human trigger at all, appropriate because staging's environment policy (Part 1) doesn't require the friction production does. repairRolloutRule defines what happens when a rollout actually fails verification: retry up to 2 times with exponential backoff (catching transient issues — a flaky health check, a momentary resource contention) before giving up and automatically rolling back to the last known-good release.

🔍 From the Trenches: A team configured automated rollback with zero retries — any single verification failure triggered an immediate rollback. A transient DNS resolution blip inside the cluster, lasting under two seconds, caused a health check to fail exactly once during a canary's 25% step, triggering a full automatic rollback of an otherwise completely healthy release. Engineers spent forty minutes investigating "why did the deploy fail" before finding the real answer buried in cluster DNS logs: nothing about the release was wrong at all. The two-levels-deep lesson: the surface symptom was a confusing rollback with no code-level cause, the immediate cause was zero retry tolerance for a genuinely transient failure class, and the underlying condition was that the automation policy had been configured to treat every verification failure as equally meaningful, with no distinction between "the new code is actually broken" and "something outside the release flaked once." Adding attempts: 2 with exponential backoff, as shown in the YAML above, fixed it without weakening the safety the rollback rule exists to provide.

Auditing and Tracking Every Deployment#

The exam guide's "auditing and tracking deployments" consideration (2.2) has a direct, concrete answer with Cloud Deploy: every Release and Rollout is itself a durable, queryable Cloud Deploy resource — gcloud deploy releases list and gcloud deploy rollouts list give a complete history with no separate logging system required, and every action (a promotion, an approval, a rollback) is captured in Cloud Audit Logs automatically, attributed to the identity that triggered it.

# "What's actually running in prod, and who approved it" — answerable
# directly, no separate deployment-tracking spreadsheet required
gcloud deploy rollouts list \
  --delivery-pipeline=shipment-api-pipeline \
  --target=prod \
  --project=meridian-cicd \
  --format="table(name, state, createTime)"

A Full Worked Pipeline: Meridian's Production Promotion Path#

Putting every piece of this chapter together — the actual sequence from a merged commit to a fully promoted production release, incorporating the automation and canary configuration from earlier sections.

# 1. Cloud Build (Part 3) finishes, pushes the image, and creates the
#    Cloud Deploy release referencing it
gcloud deploy releases create shipment-api-v42 \
  --delivery-pipeline=shipment-api-pipeline \
  --project=meridian-cicd \
  --region=us-central1 \
  --images=shipment-api=us-central1-docker.pkg.dev/meridian-cicd/meridian-images/shipment-api:a1b2c3d

# 2. Cloud Deploy automatically creates and runs the dev rollout first
#    (the pipeline's first stage, no approval or canary required)

# 3. The automation rule auto-promotes dev -> staging with no human
#    trigger, per this chapter's Automation resource

# 4. Staging's rollout runs; if it passes, prod's rollout is created
#    but PAUSED, because prod has requireApproval: true

# 5. A human with roles/clouddeploy.approver reviews and approves
gcloud deploy rollouts approve shipment-api-v42-to-prod-0001 \
  --delivery-pipeline=shipment-api-pipeline \
  --release=shipment-api-v42 \
  --target=prod \
  --project=meridian-cicd

# 6. Cloud Deploy runs the canary strategy: 25% traffic, verify,
#    50% traffic, verify, then completes to 100% -- with automatic
#    retry (2 attempts) and rollback-on-exhaustion if verification
#    ever fails, per the Automation resource's repairRolloutRule

🧪 Hands-on checkpoint: after step 6 completes, run the audit query from the previous section and confirm the Rollout history shows the full sequence — dev, staging (auto-promoted), and prod (approval-gated, canary-staged) — as one continuous, queryable record with no gaps and no separate system needed to reconstruct what happened.

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A production Target currently uses the standard deployment strategy with no approval gate. A recent bad release caused a full-traffic outage for six minutes before anyone noticed. What two independent changes would most directly address this, and why are both worth making rather than just one?

Answer: Add requireApproval: true (a human reviews before any change reaches the cluster) and switch to a canary strategy with a verify phase (limits exposure to a fraction of traffic and catches a regression automatically before it reaches 100%). They're independent and complementary: approval catches problems a human can recognize before deployment starts, while canary+verify catches problems that only manifest once real traffic hits the new version — a human approving a release doesn't guarantee the release is actually healthy under load, and a canary alone doesn't stop a change nobody should have approved in the first place from starting its rollout.

Problem 2: An Automation resource has a repairRolloutRule with attempts: 2 but no rollback block defined. What happens if both retries fail, and is this a safe default to leave as-is for a production target?

Answer: Per Cloud Deploy's automation behavior, if no rollback is configured (or configured retries are exhausted with no rollback rule), a new rollout is created to roll back to the most recently successful release on that target — rollback still happens, it's just Cloud Deploy's own default fallback behavior. Whether this is "safe to leave as-is" depends on whether that default fallback's exact behavior (target phase, verification requirements on the rollback rollout itself) matches what the team actually wants; the safer, more explicit choice is still to declare the rollback block deliberately rather than depend on undocumented default behavior nobody on the team consciously chose.

Problem 3: Meridian wants dev to auto-promote from every merged commit with zero manual steps, but wants staging to require someone to explicitly kick off promotion (without needing full production-grade approval). How would you configure this across Target and Automation resources?

Answer: Leave dev's Target with no requireApproval and give it a promoteReleaseRule in an Automation resource so every dev rollout completion automatically creates the next release for promotion. For staging, don't add a promoteReleaseRule selector targeting it (so nothing auto-promotes into it) and also don't set requireApproval: true on its Target (since that specifically gates deployment to that target with the formal approver-role mechanism) — instead, promotion into staging happens via an explicit, manually-run gcloud deploy releases promote --to-target=staging command, giving a human a deliberate trigger without the heavier formal-approval audit trail reserved for prod.

Summary and What's Next#

This chapter completed the CI/CD picture Part 3 started: Cloud Deploy's DeliveryPipeline/Target/Release/Rollout object model, Skaffold as the rendering engine underneath every promotion, the standard/canary/blue-green strategy tradeoffs, approval gates as the one deliberate human checkpoint, and automation rules that turn promotion, retry, and rollback into declared policy rather than external scripting. Meridian's production promotion path — auto-promoted staging, approval-gated and canary-staged production, with retry-then-rollback built in — is the concrete pipeline every prior chapter's design decisions (the tooling project, the environment policy differences, the fail-fast CI ordering) were ultimately building toward.

Part 5 turns to what flows through this pipeline besides the container image itself: environment-specific configuration and secrets — how Meridian keeps dev, staging, and prod configuration correctly separated without hardcoding a single value into the pipeline definition itself.