# GCP DevOps & CI/CD Platform — Part 5: Managing Environments, Configuration & Secrets > **Series:** GCP DevOps & CI/CD Platform (5 of 6) — aligned to Professional Cloud DevOps Engineer (PCDE) exam sections 1-2 > **Part 1:** `01-designing-a-devops-organization.md` — Designing a DevOps Organization > **Part 2:** `02-infrastructure-as-code-and-gitops.md` — Infrastructure as Code & GitOps > **Part 3:** `03-continuous-integration-with-cloud-build.md` — Continuous Integration with Cloud Build > **Part 4:** `04-continuous-delivery-with-cloud-deploy.md` — Continuous Delivery with Cloud Deploy > **Part 5:** This file — Managing Environments, Configuration & Secrets > **Part 6:** `06-securing-the-deployment-pipeline.md` — Securing the Deployment Pipeline & Dev Environments > **Questions:** `questions.md` ## Table of Contents 1. [The Problem: One Pipeline, Three Different Configurations](#the-problem-one-pipeline-three-different-configurations) 2. [Secret Manager: Credentials, Not Configuration](#secret-manager-credentials-not-configuration) 3. [Parameter Manager: Configuration, Not Credentials](#parameter-manager-configuration-not-credentials) 4. [Certificate Manager and Cloud KMS](#certificate-manager-and-cloud-kms) 5. [Build-Time Versus Runtime Secret Injection](#build-time-versus-runtime-secret-injection) 6. [A Decision Framework: Where Should This Value Actually Live?](#a-decision-framework-where-should-this-value-actually-live) 7. [Managing Ephemeral Environments](#managing-ephemeral-environments) 8. [Managing GKE Clusters Across an Enterprise: Fleets](#managing-gke-clusters-across-an-enterprise-fleets) 9. [Safe and Secure Patching and Upgrading Practices](#safe-and-secure-patching-and-upgrading-practices) 10. [A Full Worked Example: Meridian's Per-Environment Config Flow](#a-full-worked-example-meridians-per-environment-config-flow) 11. [Common Mistakes and Interview Traps](#common-mistakes-and-interview-traps) 12. [Worked Practice Problems](#worked-practice-problems) 13. [Summary and What's Next](#summary-and-whats-next) ## The Problem: One Pipeline, Three Different Configurations Part 4's pipeline promotes the exact same container image through `dev`, `staging`, and `prod` — that's the point, per Part 1's "build once, promote the same artifact" principle. But the *same image* still needs to behave differently in each environment: a different database connection string, a different feature-flag state, a different API key for a third-party service, different resource limits. **The exam guide's "managing multiple environments" (1.4) and "managing pipeline configuration and secrets" (2.3) sections are really one question asked twice: how does an identical artifact get the environment-specific context it needs without that context being baked into the image itself or hardcoded into the pipeline definition?** Meridian hit this concretely: `shipment-api`'s database connection string pointed at `meridian-staging`'s Cloud SQL instance in every environment for three days after a rushed deploy, because a developer had hardcoded it directly in a config file rather than reading it from anywhere environment-aware — dev traffic was quietly hitting the staging database the whole time, corrupting staging's test data with real (if low-volume) dev noise before anyone noticed the query patterns looked wrong. ```mermaid flowchart TB Image["ONE container image
(same across all environments)"] Image --> Dev["dev: dev DB, feature flags ON,
relaxed rate limits"] Image --> Staging["staging: staging DB,
flags match prod, real rate limits"] Image --> Prod["prod: prod DB, flags stable,
strict rate limits"] classDef image fill:#e5f0fa,stroke:#1d6fb8,color:#10161c classDef env fill:#f0e9fb,stroke:#6d43c0,color:#10161c class Image image class Dev,Staging,Prod env ``` **What to notice**: nothing in the image itself changes between these three boxes — every difference has to come from outside the image, injected at deploy or runtime, which is exactly the mechanism the rest of this chapter builds. ## Secret Manager: Credentials, Not Configuration Secret Manager is GCP's managed store for genuinely sensitive values — API keys, database passwords, TLS private keys, OAuth client secrets — anything where exposure is itself the incident, not just an inconvenience. Values are stored as **secret versions** (each secret can hold multiple versions, supporting rotation without downtime), encrypted at rest by default with Google-managed keys, or with a customer-managed key from Cloud KMS when a compliance requirement demands control over the encryption key itself. ```bash # Create a secret and its first version, then grant the cross-project # deployer service account from Part 1 read access -- narrowly, to # THIS secret only, not project-wide Secret Manager access gcloud secrets create shipment-api-db-password \ --project=meridian-staging \ --replication-policy=automatic echo -n "the-actual-password" | gcloud secrets versions add shipment-api-db-password \ --project=meridian-staging \ --data-file=- gcloud secrets add-iam-policy-binding shipment-api-db-password \ --project=meridian-staging \ --member="serviceAccount:meridian-deployer@meridian-cicd.iam.gserviceaccount.com" \ --role="roles/secretmanager.secretAccessor" ``` > [!TIP] > **Best Practice**: grant `roles/secretmanager.secretAccessor` on the individual secret, not at the project level. A project-level grant hands the deployer service account read access to *every* secret in that project — including ones a completely different, unrelated service owns — the same least-privilege discipline Part 1 established for cross-project IAM, applied one level more granularly here. 🔍 **From the Trenches**: A team rotated a compromised API key by creating a new secret *version*, confident the rotation was complete because `gcloud secrets versions list` showed the new version as the latest. Their application, however, had cached the secret's value in memory at startup and only re-read it on a full pod restart — the compromised key stayed live in every running pod for eleven days after the "rotation," because nobody had defined what actually triggers a re-read. The two-levels-deep lesson: the surface symptom was "the key is still being used somewhere," the immediate cause was in-memory caching with no re-read trigger, and the underlying condition was that rotation had been treated as a Secret Manager-side action alone, with no corresponding requirement on the *consuming application* to actually pick up the new version — creating a new version rotates what Secret Manager serves, not what a long-running process already holds in memory. ## Parameter Manager: Configuration, Not Credentials Parameter Manager is a newer, related-but-distinct GCP service — built as an extension of Secret Manager's underlying infrastructure, but purpose-built for **configuration** rather than credentials: database connection strings (the hostname and port, not the password), feature-flag state, environment names, port numbers, and other values every environment needs but that aren't secrets in the "exposure is the incident" sense. | Aspect | Secret Manager | Parameter Manager | |---|---|---| | What it stores | Credentials — API keys, passwords, private keys, tokens | Configuration — connection strings, feature flags, environment settings | | Format validation | None — opaque binary payload | Built-in JSON/YAML structure validation | | Sensitivity model | Everything is treated as maximally sensitive by default | Mixed — can hold sensitive and non-sensitive values, and can *reference* a Secret Manager secret inline for the sensitive parts | | Rotation urgency | Often tied to a real incident response (a leaked key) | Tied to a deploy/config-change cadence, not an incident | | GCP maturity (as of this writing) | Generally available, long-established | Preview — verify current status against Google's own release notes before depending on it for a hard production requirement | 💡 **The transferable insight**: this split mirrors AWS's Secrets Manager vs. Parameter Store distinction almost exactly — the same underlying design decision (separate sensitive credentials from general configuration, but let one reference the other for the mixed case) shows up on both clouds, because the operational reasoning behind it is provider-independent: configuration changes far more often than credentials do, and treating both identically means either over-auditing routine config changes or under-protecting real secrets. ```yaml # A Parameter Manager parameter referencing a Secret Manager secret # for its one genuinely sensitive field -- the connection string's # HOST is plain configuration, the embedded credential reference # points back to Secret Manager rather than duplicating the password database: host: "10.20.0.5" port: 5432 name: "shipment_orders" password_secret_ref: "projects/meridian-staging/secrets/shipment-api-db-password/versions/latest" ``` > [!NOTE] > Parameter Manager's preview status (at time of writing) is a real, worth-checking-current-docs detail, not a stale note to ignore — a production platform depending on a preview-tier service for something load-bearing should have a documented fallback if that service's SLA or API surface changes before general availability. ## Certificate Manager and Cloud KMS Two more pieces round out the "managing pipeline configuration and secrets" section's key-management scope (2.3): **Certificate Manager** manages TLS certificates — provisioning, renewal, and attaching them to load balancers — removing the operational burden of manually tracking certificate expiry across every environment. A certificate expiring unnoticed is a specific, recurring category of self-inflicted outage across the whole industry, and Certificate Manager's managed renewal exists specifically to remove the human "remember to renew this" step from that failure mode entirely. **Cloud KMS** is the encryption-key management layer underneath both Secret Manager's optional customer-managed encryption and any application that needs to encrypt/decrypt data directly (envelope encryption for a data store, signing a JWT, and so on) rather than relying on GCP's default encryption. For a CI/CD pipeline specifically, Cloud KMS keys also commonly protect Terraform state file encryption and sign container image provenance attestations — the latter covered in depth in Part 6's supply-chain security discussion. ```mermaid flowchart LR KMS["Cloud KMS
(key management)"] --> SM["Secret Manager
(CMEK option)"] KMS --> TFState["Terraform state
encryption"] KMS --> Sign["Image provenance
signing (Part 6)"] CM["Certificate Manager"] --> LB["Load balancer
TLS termination"] classDef kms fill:#e5f0fa,stroke:#1d6fb8,color:#10161c classDef consumer fill:#f0e9fb,stroke:#6d43c0,color:#10161c class KMS,CM kms class SM,TFState,Sign,LB consumer ``` ## Build-Time Versus Runtime Secret Injection The exam guide names this distinction explicitly (2.3), and it's a genuinely consequential design decision, not a style preference: **does a secret get baked into the build artifact itself, or fetched by the running application at startup/request time?** **Build-time injection** — a secret is read during the Cloud Build pipeline and embedded into the artifact (an environment variable baked into a container image layer, a config file copied in during the build). This is almost always the wrong default: the secret is now part of the image's layers, retrievable by anyone who can pull that image from Artifact Registry, and rotating it means rebuilding and redeploying the entire artifact. **Runtime injection** — the application reads the secret at startup or on-demand, directly from Secret Manager (via the client library, or via a mounted volume in GKE using the Secret Manager CSI driver). The image itself never contains the secret value; only a reference (a secret name/path) does, and rotation only requires the running process to re-read, not a full rebuild. ```mermaid sequenceDiagram participant Build as Cloud Build participant SM as Secret Manager participant Image as Container Image participant Pod as Running Pod rect rgb(251, 232, 230) Note over Build,Image: Build-time injection (avoid for real secrets) Build->>SM: Fetch secret value Build->>Image: Bake value into image layer end rect rgb(229, 245, 234) Note over Pod,SM: Runtime injection (the safer default) Pod->>SM: Fetch secret value at startup
(via Workload Identity, no static key) SM-->>Pod: Value returned, never touches the image end ``` **What to notice**: in the build-time path, the secret's value physically exists inside a stored artifact indefinitely, retrievable from any image pull, forever — in the runtime path, it exists only in the running process's memory, fetched fresh via an identity (Part 1's Workload Identity Federation pattern) that leaves no static credential anywhere to steal. > [!WARNING] > An environment variable set via `ENV` in a Dockerfile, or via `--build-arg`, is **build-time injection** even if it looks like ordinary runtime configuration — both land in the image's build history and are recoverable with `docker history` or by inspecting the image's layers, regardless of whether the final running container ever prints the value anywhere. This is a common, easy-to-miss version of the mistake this section warns against. ## A Decision Framework: Where Should This Value Actually Live? Between Kubernetes-native `ConfigMap`s, plain environment variables, Parameter Manager, and Secret Manager, a team can genuinely reach for the wrong mechanism out of habit rather than deliberate choice. This table collects every option this chapter has touched into one practitioner-facing decision point. | Value | Right home | Why not the others | |---|---|---| | A database password | Secret Manager | A `ConfigMap` and plain env var are both stored unencrypted-at-rest by default and visible to anyone with `kubectl get` access to the namespace | | A database hostname/port | Parameter Manager (or a `ConfigMap` if the value never needs cross-service reuse) | Secret Manager's audit/access overhead is unwarranted for a non-sensitive value; a plain env var baked at build time breaks per-environment portability | | A feature-flag boolean | Parameter Manager | Needs to change without a redeploy in many designs, and format validation (JSON/YAML) catches a malformed flag payload before it reaches a pod | | A container's log level (`debug`/`info`) | A `ConfigMap`, referenced as an env var | Genuinely low-stakes, changes rarely, doesn't need Parameter Manager's validation or audit trail | | A TLS certificate/private key | Certificate Manager (cert) + Secret Manager (private key, if self-managed) | A `ConfigMap` has no encryption-at-rest guarantee suited to key material, and certs specifically benefit from Certificate Manager's automated renewal | | A third-party API key | Secret Manager | Exposure is a real incident regardless of which environment it's for | 💡 **The transferable insight**: the single question that resolves almost every case in this table is **"if this value leaked in a log line or a misconfigured RBAC grant, would that be an inconvenience or an incident?"** An inconvenience-tier value belongs in a `ConfigMap` or Parameter Manager; an incident-tier value belongs in Secret Manager, full stop — the mechanism should match the actual consequence of exposure, not habit or whatever was already wired up for a different value. ## Managing Ephemeral Environments Beyond the three persistent environments (dev/staging/prod) this course has focused on, the exam guide separately calls out **ephemeral environments** — a temporary, full or partial environment spun up for one specific purpose (testing one pull request in isolation, a load test, a demo) and torn down afterward. This solves a real, specific problem persistent staging can't: two feature branches both needing to modify the same database schema can't safely share one persistent staging environment at the same time, but each can get its own throwaway environment with no conflict. ```yaml # A Cloud Build trigger creating a namespace-scoped ephemeral # environment per pull request, torn down when the PR closes -- # using the SAME Cloud Deploy pipeline shape from Part 4, just # targeting a short-lived namespace instead of a persistent cluster steps: - id: 'create-ephemeral-namespace' name: 'gcr.io/cloud-builders/kubectl' args: ['create', 'namespace', 'pr-${_PR_NUMBER}'] env: - 'CLOUDSDK_COMPUTE_ZONE=us-central1' - 'CLOUDSDK_CONTAINER_CLUSTER=meridian-dev' ``` > [!TIP] > **Best Practice**: attach a hard time-to-live to every ephemeral environment (a scheduled cleanup job, or a Cloud Deploy automation rule tied to the source PR's closure) rather than relying on someone remembering to tear it down manually. An "ephemeral" environment nobody actively deletes is just a permanent environment with a misleading name, quietly accumulating cost and configuration drift indefinitely. ## Managing GKE Clusters Across an Enterprise: Fleets As Meridian's cluster count grows past a handful, managing each GKE cluster's configuration, policy, and upgrade schedule individually stops scaling — this is exactly what GKE **fleets** solve. A fleet is a logical grouping of clusters (and other resources) managed together through the Fleet API, letting policy, Config Sync (Part 2), and monitoring configuration apply once across every member cluster rather than once per cluster by hand. > [!NOTE] > As of 2026, the fleet-management capabilities that used to require the paid Anthos/GKE Enterprise tier — the Fleet API itself, Config Sync, Policy Controller, and Connect Gateway — moved into base GKE at no separate license cost. Verify this against current GKE documentation if working from older material, since this is a genuine, relatively recent pricing/tiering change. ```bash # Register Meridian's three environment clusters into one fleet, # so a single Config Sync RootSync (Part 2) and a single fleet-wide # policy can target all three instead of three separate configs gcloud container fleet memberships register meridian-dev-membership \ --gke-cluster=us-central1/meridian-dev \ --project=meridian-fleet-host gcloud container fleet memberships register meridian-staging-membership \ --gke-cluster=us-central1/meridian-staging \ --project=meridian-fleet-host ``` ## Safe and Secure Patching and Upgrading Practices GKE's **release channels** — Rapid, Regular, and Stable — subscribe a cluster to an automated upgrade cadence matched to a team's real risk tolerance, rather than leaving version upgrades as a manually-triggered, easily-postponed chore. | Channel | Upgrade cadence | Best fit | |---|---|---| | Rapid | Newest GKE versions, shortly after release | A dedicated test cluster validating new Kubernetes features early, never a production workload | | Regular | A balanced cadence, versions battle-tested but not bleeding-edge | Most production clusters — Meridian's own default for `staging` and `prod` | | Stable | The most conservative, longest-validated versions | Workloads with the lowest tolerance for any upgrade-related surprise | **Maintenance windows** define *when* GKE is allowed to perform automated upgrades (a recurring low-traffic period), and **maintenance exclusions** temporarily block them entirely — for up to 90 days — during a period where any disruption risk is unacceptable (a peak shopping season, a planned major product launch). ```yaml # A maintenance exclusion blocking upgrades during Meridian's own # highest-traffic period -- their logistics customers' fiscal # year-end shipping surge, when an upgrade-triggered node drain # would be the worst possible time for even brief disruption maintenancePolicy: window: maintenanceExclusions: - name: "fiscal-year-end-freeze" window: startTime: "2027-03-25T00:00:00Z" endTime: "2027-04-05T00:00:00Z" ``` 🚨 **Incident-critical action**: set a maintenance exclusion *before* a known high-risk traffic period begins, not reactively after an upgrade has already started causing disruption — once GKE has begun a control-plane or node upgrade, an exclusion added after the fact does not retroactively pause work already in progress. ## A Full Worked Example: Meridian's Per-Environment Config Flow Bringing this chapter's pieces together — the actual mechanism Priya's team built to fix the hardcoded-connection-string incident from this chapter's opening. ```mermaid sequenceDiagram participant CD as Cloud Deploy Rollout participant PM as Parameter Manager participant SM as Secret Manager participant Pod as shipment-api Pod CD->>Pod: Deploy manifest referencing
environment-specific ConfigMap Pod->>PM: Read connection config
(host, port, db name) PM-->>Pod: Environment-correct values
(staging host, staging port) Pod->>SM: Read db password
via Workload Identity SM-->>Pod: Password, scoped to THIS
environment's secret only Pod->>Pod: Application starts,
connects to correct database ``` The application code itself never branches on which environment it's running in — it always reads the same two logical values (connection config, credential) from the same two services, and it's the *deploy-time manifest* (rendered per-`Target` by Skaffold, per Part 4) that determines which specific Parameter Manager parameter and Secret Manager secret those reads resolve to. This is the concrete fix: environment-awareness lives entirely in deploy-time configuration, never in application logic or a hardcoded value. 🧪 **Hands-on checkpoint**: confirm your own service's configuration-reading code takes a parameter/secret *name* as input (from an environment variable or mounted file path that Skaffold's per-environment profile sets) rather than a hardcoded resource identifier — that one indirection is what makes the same image genuinely portable across every environment. ## Common Mistakes and Interview Traps | Mistake | Why it's wrong | What to say/do instead | |---|---|---| | Hardcoding a database host/credential in application config | Breaks the "one image, many environments" model; caused Meridian's real cross-environment data incident | Read connection details from Parameter Manager, credentials from Secret Manager, resolved per-environment at deploy time | | Baking a secret into a Docker image via `ENV`/`--build-arg` | Recoverable from `docker history` regardless of runtime behavior — this is build-time injection even if it doesn't look like it | Use runtime injection: read from Secret Manager at startup via Workload Identity | | Assuming a new Secret Manager version rotates a running process's cached value | A long-running process holding the value in memory won't see the new version until it re-reads | Design applications to re-read on a defined trigger (periodic refresh, SIGHUP, or a restart as part of rotation) | | Leaving an "ephemeral" environment with no automatic teardown | Becomes a permanent environment with a misleading name, silently accruing cost and drift | Attach a hard TTL or an automation rule tied to the source PR's lifecycle | | Granting a secret-access IAM binding at the project level | Grants access to every secret in the project, not just the one the service actually needs | Bind `roles/secretmanager.secretAccessor` on the individual secret | | Adding a maintenance exclusion only after an upgrade has already started causing disruption | Exclusions don't pause work already in progress | Set exclusions ahead of known high-risk traffic windows | ## Worked Practice Problems **Problem 1**: A service's Secret Manager secret was rotated (a new version added) to respond to a suspected credential leak, but three days later logs show requests still authenticating with the old, supposedly-revoked value. What's the most likely explanation, and what fixes it going forward? *Answer*: The most likely explanation is exactly this chapter's From-the-Trenches scenario — the running application cached the secret's value in memory at startup and has no mechanism to re-read a newer version without a restart. Creating a new secret version changes what Secret Manager *serves*, not what an already-running process already holds. The fix going forward is designing the application (or its deployment) to actually re-read on rotation — a scheduled restart tied to rotation events, a SIGHUP-triggered reload, or a sidecar that watches for version changes and restarts the main process. **Problem 2**: A teammate proposes storing a service's feature-flag configuration in Secret Manager "since it's already set up and working." What's the concern with this choice, even though it would technically function? *Answer*: Feature-flag configuration is exactly the kind of non-sensitive, frequently-changing configuration Parameter Manager is purpose-built for — Secret Manager's audit and access-control model is designed around the assumption that every read is sensitive and every change is a security-relevant event, which adds unnecessary audit noise and access-control overhead for a value that changes routinely and carries no real exposure risk if seen. It would work technically, but it mismatches the tool to the actual sensitivity of the data, the same category of mismatch this chapter's comparison table exists to prevent. **Problem 3**: A `docker build` command uses `--build-arg DB_PASSWORD=$SECRET_VALUE` to pass a database password into the image build, and the Dockerfile never actually prints or logs the value anywhere. Is this safe, and why or why not? *Answer*: No — this is build-time secret injection regardless of whether the Dockerfile ever visibly uses the value in a log or printed output. Docker build arguments are recorded in the image's build history and are recoverable via `docker history` or by inspecting the image's layer metadata by anyone who can pull the image, independent of what the running container prints. The fix is runtime injection: never pass a real secret as a build argument, and have the running application fetch it from Secret Manager directly at startup. ## Summary and What's Next This chapter solved the "one artifact, many environments" problem Part 4's pipeline left open: Secret Manager for genuine credentials, Parameter Manager for general configuration (with the ability to reference a secret inline for its sensitive fields), Certificate Manager and Cloud KMS rounding out the key-management surface, and the build-time-versus-runtime injection distinction as the single most consequential secret-handling decision a pipeline makes. Ephemeral environments, GKE fleets, and release-channel-driven patching extended the same "manage many environments/clusters consistently, not by hand, one at a time" principle from configuration into infrastructure lifecycle itself. **Part 6**, the final chapter of this course, turns to securing the pipeline itself: vulnerability scanning, Binary Authorization, the SLSA supply-chain framework, and hardening the developer environments (Cloud Workstations, Cloud Shell) that feed everything this course has built.