HashiCorp Vault

What it is and where it fits 🎯#

HashiCorp Vault is the most widely deployed, tool-agnostic secrets management platform — the dedicated secrets manager this series' Secrets Management & IAM chapter builds its entire architecture discussion around. Instead of a static database password baked into an environment variable or config file forever, an application authenticates to Vault at runtime, Vault checks that identity against a policy, and hands back exactly the secret(s) that policy allows — logged, auditable, and revocable at any point. Vault's standout capability, and the reason it's worth learning at an architectural level rather than as a product name to recognize, is dynamic secrets: for supported backends (databases, cloud IAM, PKI), Vault doesn't just store a secret someone else created — it generates a brand-new, unique, automatically-expiring credential on demand, per requesting workload.

Three tools in this same cheat-sheet series solve adjacent but distinct problems, and knowing the boundary matters for an interview or a real design review: Vault is the runtime secrets source of truth — an application calls its API directly, or gets a token some other way. SOPS (its own cheat sheet) takes the opposite approach — it encrypts a secret file so it's safe to commit to git and decrypts it at deploy time, with no server to run. External Secrets Operator (its own cheat sheet) sits between the two for Kubernetes specifically: it runs inside the cluster and continuously syncs a secret from Vault (or AWS Secrets Manager, or any supported backend) into a native Kubernetes Secret, so pods never call Vault's API directly at all.

How a client actually gets a secret from Vault#

The token issued in step 3 is what every subsequent call actually authorizes against — it's short-lived and tied to a policy, never a long-lived static credential a leaked laptop or log file could expose indefinitely.

Installation#

# Debian/Ubuntu — HashiCorp's official apt repository
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | \
  sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install vault

# Manual binary (any platform) — download a pinned version from releases.hashicorp.com/vault
curl -O https://releases.hashicorp.com/vault/1.19.0/vault_1.19.0_linux_amd64.zip
unzip vault_1.19.0_linux_amd64.zip && sudo mv vault /usr/local/bin/

# macOS
brew tap hashicorp/tap && brew install hashicorp/tap/vault

vault version

Core concepts#

Starting and checking a server#

vault server -config=/etc/vault.d/vault.hcl    # real server, using a config file (see below)
vault status                                    # sealed/unsealed, HA state, storage type
vault operator unseal <unseal-key>               # supply one key of the configured quorum, repeat until unsealed
vault login <root-or-personal-token>             # authenticate the CLI itself for the current shell

KV secrets engine (static key/value)#

vault secrets enable -path=secret kv-v2                       # mount a KV v2 engine at secret/ (v2 keeps version history)
vault kv put secret/checkout/db-credentials username=svc password="S3cr3t!"
vault kv get secret/checkout/db-credentials                     # latest version, table output
vault kv get -field=password secret/checkout/db-credentials     # a single field, for scripting
vault kv get -version=2 secret/checkout/db-credentials           # a specific historical version (KV v2 only)
vault kv patch secret/checkout/db-credentials password="N3wP4ss!" # update one field without clobbering the rest
vault kv metadata get secret/checkout/db-credentials              # version history, without revealing values
vault kv delete secret/checkout/db-credentials                    # soft-delete the latest version (recoverable)
vault kv undelete -versions=3 secret/checkout/db-credentials      # restore a soft-deleted version
vault kv destroy -versions=3 secret/checkout/db-credentials       # permanently destroy a specific version's data
vault kv list secret/checkout/                                     # list keys under a path

kv put fully overwrites all fields at that path unless you use kv patch — a common early mistake is running kv put a second time to add one field and accidentally deleting every other field that was there.

Database secrets engine — dynamic secrets in practice#

This is the exact mechanism the tutorial's checkout-service example builds toward: instead of one shared Postgres password every instance uses forever, Vault creates a brand-new, unique database user per request, with a lease that auto-expires.

vault secrets enable database                     # mount the database secrets engine at database/

# Tell Vault how to connect, and which Vault roles are allowed to use this connection
vault write database/config/checkout-postgres \
    plugin_name=postgresql-database-plugin \
    allowed_roles="checkout-readonly" \
    connection_url="postgresql://{{username}}:{{password}}@postgres:5432/checkout?sslmode=disable" \
    username="vault-admin" \
    password="admin-bootstrap-password"

# Define a Vault role: a TEMPLATE for creating short-lived SQL users, not a single fixed user
vault write database/roles/checkout-readonly \
    db_name=checkout-postgres \
    creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
      GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
    default_ttl=1h \
    max_ttl=24h

vault read database/creds/checkout-readonly         # generates a brand-new, unique username/password RIGHT NOW
vault lease renew database/creds/checkout-readonly/<lease_id>   # extend before the TTL expires, up to max_ttl
vault lease revoke database/creds/checkout-readonly/<lease_id>  # revoke immediately — the SQL user is dropped

Sample vault read database/creds/checkout-readonly output (representative shape — the actual username is auto-generated and unique on every call, so it will never repeat):

Policies — least-privilege access to secret paths#

# checkout-policy.hcl
path "secret/data/checkout/*" {
  capabilities = ["read", "list"]
}
path "database/creds/checkout-readonly" {
  capabilities = ["read"]
}

vault policy write checkout-policy checkout-policy.hcl
vault policy read checkout-policy
vault policy list

A policy that only grants read/list on the exact paths a workload needs — nothing broader — is Vault's version of the same least-privilege principle this series applies repeatedly to Kubernetes RBAC and IAM roles: the blast radius of a compromised token is bounded by what its policy actually allows.

Kubernetes auth method — no static Vault credential in any pod#

vault auth enable kubernetes
vault write auth/kubernetes/config \
    kubernetes_host="https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT"

vault write auth/kubernetes/role/checkout-service \
    bound_service_account_names=checkout-service \
    bound_service_account_namespaces=default \
    policies=checkout-policy \
    ttl=1h

# From inside the pod (its own ServiceAccount token is the proof of identity):
vault write auth/kubernetes/login role=checkout-service \
    jwt=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)

This closes the same "secret zero" loop the tutorial names explicitly: the pod never stores any Vault credential at all — it authenticates using an identity Kubernetes itself already issues and vouches for.

JWT/OIDC auth method — federating CI/CD platforms directly#

vault auth enable jwt
vault write auth/jwt/config \
    oidc_discovery_url="https://token.actions.githubusercontent.com" \
    bound_issuer="https://token.actions.githubusercontent.com"

vault write auth/jwt/role/github-deploy \
    role_type="jwt" \
    bound_audiences="https://github.com/my-org" \
    bound_claims='{"repository":"my-org/checkout-service","ref":"refs/heads/main"}' \
    user_claim="actor" \
    policies="checkout-policy" \
    ttl=15m

# Inside a GitHub Actions job with id-token: write permission:
VAULT_TOKEN=$(vault write -field=token auth/jwt/login \
    role=github-deploy jwt=$ACTIONS_ID_TOKEN_REQUEST_TOKEN)

bound_claims is the line doing the real security work — a validly-signed token from a different repository or branch still fails this check and is rejected, exactly like an IAM role trust-policy condition scoping which OIDC subject can assume it.

Token management#

vault token create -policy=checkout-policy -ttl=1h    # issue a scoped, time-limited token by hand
vault token lookup <token>                              # inspect a token's policies, TTL, and metadata
vault token renew <token>
vault token revoke <token>                              # immediately invalidate — and everything it leased

Audit devices — who accessed what, when#

vault audit enable file file_path=/var/log/vault_audit.log
vault audit list

Config file format — a minimal production server config#

# vault.hcl
storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-node-1"
}

listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/etc/vault.d/tls/vault-cert.pem"
  tls_key_file  = "/etc/vault.d/tls/vault-key.pem"
}

seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "alias/vault-unseal-key"
}

api_addr     = "https://vault.internal.example.com:8200"
cluster_addr = "https://vault-node-1.internal.example.com:8201"
ui           = true

The seal "awskms" stanza is what auto-unseal actually is in practice — Vault calls out to a cloud KMS to decrypt its own master key on startup instead of requiring a human to type in unseal key shares every time the process restarts, while still keeping the underlying storage itself independently encrypted.

Real-world scenario: killing a shared static database password#

checkout-service currently has one hardcoded Postgres password, identical across every running instance, sitting in config.py — the exact scenario the tutorial's secrets-management chapter opens with. Fixing it:

• Enable the database secrets engine and configure the connection (database/config/checkout-postgres).
• Define a Vault role with the minimal SQL grants the service actually needs (SELECT only, not superuser) and a default_ttl/max_ttl pair that matches how long a request cycle realistically runs.
• Enable the Kubernetes auth method and bind a role to checkout-service's exact ServiceAccount and namespace.
• Update the application to call vault read database/creds/checkout-readonly at startup (or via a sidecar like Vault Agent) instead of reading a password from an environment variable.
• Revoke and rotate the old static database user entirely — leaving it active defeats the whole point.
• Enable an audit device so every credential issuance is logged.

CI/CD integration recipe — GitHub Actions, OIDC, no stored Vault token#

# .github/workflows/deploy.yml
name: Deploy checkout-service
on:
  push:
    branches: [main]

permissions:
  id-token: write   # required to request the GitHub OIDC token
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - name: Fetch DB credentials from Vault via JWT auth
        run: |
          VAULT_TOKEN=$(vault write -field=token auth/jwt/login \
            role=github-deploy jwt=$ACTIONS_ID_TOKEN_REQUEST_TOKEN)
          export VAULT_TOKEN
          DB_PASSWORD=$(vault kv get -field=password secret/checkout/db-credentials)
          echo "::add-mask::$DB_PASSWORD"
        env:
          VAULT_ADDR: https://vault.internal.example.com:8200

::add-mask:: matters here specifically because this is the one place the secret briefly exists as a shell variable inside CI logs — without masking it, a stray echo or failed-step stack trace elsewhere in the job could leak it straight into the build log.

Common pitfalls#

• Treating a root token as a day-to-day credential. The root token can do anything, including rewriting every policy — generate it once for initial setup, then revoke it and operate through scoped tokens/auth methods for everything else.
• Forgetting max_ttl. A renewable lease with no practical ceiling can end up effectively long-lived, quietly defeating the point of "dynamic" secrets.
• Running -dev mode anywhere reachable from outside a laptop. See the warning under Installation.
• kv put silently overwriting fields. Use kv patch for a partial update on KV v2.
• No auto-unseal in production. A server that restarts and needs a human to manually supply unseal keys is an availability risk during an incident, exactly when you can least afford a manual step.

When to reach for something else#

For secrets that need to be safely committed to a GitOps repository rather than fetched from a live API at runtime, reach for SOPS instead (its own cheat sheet) — no server to run or keep available. For syncing Vault-held secrets into native Kubernetes Secret objects without every pod needing its own Vault client logic, reach for External Secrets Operator (its own cheat sheet), which wraps this exact Vault API underneath a Kubernetes-native CRD. If the team is fully committed to one cloud provider and doesn't need dynamic secrets' breadth of supported backends, a fully-managed option like AWS Secrets Manager trades some flexibility for zero operational overhead — see the comparison table in Part 4 of the DevSecOps tutorial.