Managing Environments, Configuration & Secrets

Table of Contents#

1. The Problem: One Pipeline, Three Different Configurations
2. Secret Manager: Credentials, Not Configuration
3. Parameter Manager: Configuration, Not Credentials
4. Certificate Manager and Cloud KMS
5. Build-Time Versus Runtime Secret Injection
6. A Decision Framework: Where Should This Value Actually Live?
7. Managing Ephemeral Environments
8. Managing GKE Clusters Across an Enterprise: Fleets
9. Safe and Secure Patching and Upgrading Practices
10. A Full Worked Example: Meridian's Per-Environment Config Flow
11. Common Mistakes and Interview Traps
12. Worked Practice Problems
13. Summary and What's Next

The Problem: One Pipeline, Three Different Configurations#

Part 4's pipeline promotes the exact same container image through dev, staging, and prod — that's the point, per Part 1's "build once, promote the same artifact" principle. But the same image still needs to behave differently in each environment: a different database connection string, a different feature-flag state, a different API key for a third-party service, different resource limits. The exam guide's "managing multiple environments" (1.4) and "managing pipeline configuration and secrets" (2.3) sections are really one question asked twice: how does an identical artifact get the environment-specific context it needs without that context being baked into the image itself or hardcoded into the pipeline definition?

Meridian hit this concretely: shipment-api's database connection string pointed at meridian-staging's Cloud SQL instance in every environment for three days after a rushed deploy, because a developer had hardcoded it directly in a config file rather than reading it from anywhere environment-aware — dev traffic was quietly hitting the staging database the whole time, corrupting staging's test data with real (if low-volume) dev noise before anyone noticed the query patterns looked wrong.

What to notice: nothing in the image itself changes between these three boxes — every difference has to come from outside the image, injected at deploy or runtime, which is exactly the mechanism the rest of this chapter builds.

Secret Manager: Credentials, Not Configuration#

Secret Manager is GCP's managed store for genuinely sensitive values — API keys, database passwords, TLS private keys, OAuth client secrets — anything where exposure is itself the incident, not just an inconvenience. Values are stored as secret versions (each secret can hold multiple versions, supporting rotation without downtime), encrypted at rest by default with Google-managed keys, or with a customer-managed key from Cloud KMS when a compliance requirement demands control over the encryption key itself.

# Create a secret and its first version, then grant the cross-project
# deployer service account from Part 1 read access -- narrowly, to
# THIS secret only, not project-wide Secret Manager access
gcloud secrets create shipment-api-db-password \
  --project=meridian-staging \
  --replication-policy=automatic

echo -n "the-actual-password" | gcloud secrets versions add shipment-api-db-password \
  --project=meridian-staging \
  --data-file=-

gcloud secrets add-iam-policy-binding shipment-api-db-password \
  --project=meridian-staging \
  --member="serviceAccount:meridian-deployer@meridian-cicd.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

🔍 From the Trenches: A team rotated a compromised API key by creating a new secret version, confident the rotation was complete because gcloud secrets versions list showed the new version as the latest. Their application, however, had cached the secret's value in memory at startup and only re-read it on a full pod restart — the compromised key stayed live in every running pod for eleven days after the "rotation," because nobody had defined what actually triggers a re-read. The two-levels-deep lesson: the surface symptom was "the key is still being used somewhere," the immediate cause was in-memory caching with no re-read trigger, and the underlying condition was that rotation had been treated as a Secret Manager-side action alone, with no corresponding requirement on the consuming application to actually pick up the new version — creating a new version rotates what Secret Manager serves, not what a long-running process already holds in memory.

Parameter Manager: Configuration, Not Credentials#

Parameter Manager is a newer, related-but-distinct GCP service — built as an extension of Secret Manager's underlying infrastructure, but purpose-built for configuration rather than credentials: database connection strings (the hostname and port, not the password), feature-flag state, environment names, port numbers, and other values every environment needs but that aren't secrets in the "exposure is the incident" sense.

💡 The transferable insight: this split mirrors AWS's Secrets Manager vs. Parameter Store distinction almost exactly — the same underlying design decision (separate sensitive credentials from general configuration, but let one reference the other for the mixed case) shows up on both clouds, because the operational reasoning behind it is provider-independent: configuration changes far more often than credentials do, and treating both identically means either over-auditing routine config changes or under-protecting real secrets.

# A Parameter Manager parameter referencing a Secret Manager secret
# for its one genuinely sensitive field -- the connection string's
# HOST is plain configuration, the embedded credential reference
# points back to Secret Manager rather than duplicating the password
database:
  host: "10.20.0.5"
  port: 5432
  name: "shipment_orders"
  password_secret_ref: "projects/meridian-staging/secrets/shipment-api-db-password/versions/latest"

Certificate Manager and Cloud KMS#

Two more pieces round out the "managing pipeline configuration and secrets" section's key-management scope (2.3):

Certificate Manager manages TLS certificates — provisioning, renewal, and attaching them to load balancers — removing the operational burden of manually tracking certificate expiry across every environment. A certificate expiring unnoticed is a specific, recurring category of self-inflicted outage across the whole industry, and Certificate Manager's managed renewal exists specifically to remove the human "remember to renew this" step from that failure mode entirely.

Cloud KMS is the encryption-key management layer underneath both Secret Manager's optional customer-managed encryption and any application that needs to encrypt/decrypt data directly (envelope encryption for a data store, signing a JWT, and so on) rather than relying on GCP's default encryption. For a CI/CD pipeline specifically, Cloud KMS keys also commonly protect Terraform state file encryption and sign container image provenance attestations — the latter covered in depth in Part 6's supply-chain security discussion.

Build-Time Versus Runtime Secret Injection#

The exam guide names this distinction explicitly (2.3), and it's a genuinely consequential design decision, not a style preference: does a secret get baked into the build artifact itself, or fetched by the running application at startup/request time?

Build-time injection — a secret is read during the Cloud Build pipeline and embedded into the artifact (an environment variable baked into a container image layer, a config file copied in during the build). This is almost always the wrong default: the secret is now part of the image's layers, retrievable by anyone who can pull that image from Artifact Registry, and rotating it means rebuilding and redeploying the entire artifact.

Runtime injection — the application reads the secret at startup or on-demand, directly from Secret Manager (via the client library, or via a mounted volume in GKE using the Secret Manager CSI driver). The image itself never contains the secret value; only a reference (a secret name/path) does, and rotation only requires the running process to re-read, not a full rebuild.

What to notice: in the build-time path, the secret's value physically exists inside a stored artifact indefinitely, retrievable from any image pull, forever — in the runtime path, it exists only in the running process's memory, fetched fresh via an identity (Part 1's Workload Identity Federation pattern) that leaves no static credential anywhere to steal.

A Decision Framework: Where Should This Value Actually Live?#

Between Kubernetes-native ConfigMaps, plain environment variables, Parameter Manager, and Secret Manager, a team can genuinely reach for the wrong mechanism out of habit rather than deliberate choice. This table collects every option this chapter has touched into one practitioner-facing decision point.

💡 The transferable insight: the single question that resolves almost every case in this table is "if this value leaked in a log line or a misconfigured RBAC grant, would that be an inconvenience or an incident?" An inconvenience-tier value belongs in a ConfigMap or Parameter Manager; an incident-tier value belongs in Secret Manager, full stop — the mechanism should match the actual consequence of exposure, not habit or whatever was already wired up for a different value.

Managing Ephemeral Environments#

Beyond the three persistent environments (dev/staging/prod) this course has focused on, the exam guide separately calls out ephemeral environments — a temporary, full or partial environment spun up for one specific purpose (testing one pull request in isolation, a load test, a demo) and torn down afterward. This solves a real, specific problem persistent staging can't: two feature branches both needing to modify the same database schema can't safely share one persistent staging environment at the same time, but each can get its own throwaway environment with no conflict.

# A Cloud Build trigger creating a namespace-scoped ephemeral
# environment per pull request, torn down when the PR closes --
# using the SAME Cloud Deploy pipeline shape from Part 4, just
# targeting a short-lived namespace instead of a persistent cluster
steps:
  - id: 'create-ephemeral-namespace'
    name: 'gcr.io/cloud-builders/kubectl'
    args: ['create', 'namespace', 'pr-${_PR_NUMBER}']
    env:
      - 'CLOUDSDK_COMPUTE_ZONE=us-central1'
      - 'CLOUDSDK_CONTAINER_CLUSTER=meridian-dev'

Managing GKE Clusters Across an Enterprise: Fleets#

As Meridian's cluster count grows past a handful, managing each GKE cluster's configuration, policy, and upgrade schedule individually stops scaling — this is exactly what GKE fleets solve. A fleet is a logical grouping of clusters (and other resources) managed together through the Fleet API, letting policy, Config Sync (Part 2), and monitoring configuration apply once across every member cluster rather than once per cluster by hand.

# Register Meridian's three environment clusters into one fleet,
# so a single Config Sync RootSync (Part 2) and a single fleet-wide
# policy can target all three instead of three separate configs
gcloud container fleet memberships register meridian-dev-membership \
  --gke-cluster=us-central1/meridian-dev \
  --project=meridian-fleet-host

gcloud container fleet memberships register meridian-staging-membership \
  --gke-cluster=us-central1/meridian-staging \
  --project=meridian-fleet-host

Safe and Secure Patching and Upgrading Practices#

GKE's release channels — Rapid, Regular, and Stable — subscribe a cluster to an automated upgrade cadence matched to a team's real risk tolerance, rather than leaving version upgrades as a manually-triggered, easily-postponed chore.

Maintenance windows define when GKE is allowed to perform automated upgrades (a recurring low-traffic period), and maintenance exclusions temporarily block them entirely — for up to 90 days — during a period where any disruption risk is unacceptable (a peak shopping season, a planned major product launch).

# A maintenance exclusion blocking upgrades during Meridian's own
# highest-traffic period -- their logistics customers' fiscal
# year-end shipping surge, when an upgrade-triggered node drain
# would be the worst possible time for even brief disruption
maintenancePolicy:
  window:
    maintenanceExclusions:
      - name: "fiscal-year-end-freeze"
        window:
          startTime: "2027-03-25T00:00:00Z"
          endTime: "2027-04-05T00:00:00Z"

🚨 Incident-critical action: set a maintenance exclusion before a known high-risk traffic period begins, not reactively after an upgrade has already started causing disruption — once GKE has begun a control-plane or node upgrade, an exclusion added after the fact does not retroactively pause work already in progress.

A Full Worked Example: Meridian's Per-Environment Config Flow#

Bringing this chapter's pieces together — the actual mechanism Priya's team built to fix the hardcoded-connection-string incident from this chapter's opening.

The application code itself never branches on which environment it's running in — it always reads the same two logical values (connection config, credential) from the same two services, and it's the deploy-time manifest (rendered per-Target by Skaffold, per Part 4) that determines which specific Parameter Manager parameter and Secret Manager secret those reads resolve to. This is the concrete fix: environment-awareness lives entirely in deploy-time configuration, never in application logic or a hardcoded value.

🧪 Hands-on checkpoint: confirm your own service's configuration-reading code takes a parameter/secret name as input (from an environment variable or mounted file path that Skaffold's per-environment profile sets) rather than a hardcoded resource identifier — that one indirection is what makes the same image genuinely portable across every environment.

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A service's Secret Manager secret was rotated (a new version added) to respond to a suspected credential leak, but three days later logs show requests still authenticating with the old, supposedly-revoked value. What's the most likely explanation, and what fixes it going forward?

Answer: The most likely explanation is exactly this chapter's From-the-Trenches scenario — the running application cached the secret's value in memory at startup and has no mechanism to re-read a newer version without a restart. Creating a new secret version changes what Secret Manager serves, not what an already-running process already holds. The fix going forward is designing the application (or its deployment) to actually re-read on rotation — a scheduled restart tied to rotation events, a SIGHUP-triggered reload, or a sidecar that watches for version changes and restarts the main process.

Problem 2: A teammate proposes storing a service's feature-flag configuration in Secret Manager "since it's already set up and working." What's the concern with this choice, even though it would technically function?

Answer: Feature-flag configuration is exactly the kind of non-sensitive, frequently-changing configuration Parameter Manager is purpose-built for — Secret Manager's audit and access-control model is designed around the assumption that every read is sensitive and every change is a security-relevant event, which adds unnecessary audit noise and access-control overhead for a value that changes routinely and carries no real exposure risk if seen. It would work technically, but it mismatches the tool to the actual sensitivity of the data, the same category of mismatch this chapter's comparison table exists to prevent.

Problem 3: A docker build command uses --build-arg DB_PASSWORD=$SECRET_VALUE to pass a database password into the image build, and the Dockerfile never actually prints or logs the value anywhere. Is this safe, and why or why not?

Answer: No — this is build-time secret injection regardless of whether the Dockerfile ever visibly uses the value in a log or printed output. Docker build arguments are recorded in the image's build history and are recoverable via docker history or by inspecting the image's layer metadata by anyone who can pull the image, independent of what the running container prints. The fix is runtime injection: never pass a real secret as a build argument, and have the running application fetch it from Secret Manager directly at startup.

Summary and What's Next#

This chapter solved the "one artifact, many environments" problem Part 4's pipeline left open: Secret Manager for genuine credentials, Parameter Manager for general configuration (with the ability to reference a secret inline for its sensitive fields), Certificate Manager and Cloud KMS rounding out the key-management surface, and the build-time-versus-runtime injection distinction as the single most consequential secret-handling decision a pipeline makes. Ephemeral environments, GKE fleets, and release-channel-driven patching extended the same "manage many environments/clusters consistently, not by hand, one at a time" principle from configuration into infrastructure lifecycle itself.

Part 6, the final chapter of this course, turns to securing the pipeline itself: vulnerability scanning, Binary Authorization, the SLSA supply-chain framework, and hardening the developer environments (Cloud Workstations, Cloud Shell) that feed everything this course has built.