Storage: S3, EBS & EFS

What it is and where it fits#

AWS's storage services split along a fundamental line the CLI's own command structure mirrors: S3 is object storage (flat key-value, accessed over HTTPS, no filesystem semantics), while EBS and EFS are block and file storage respectively — the kind of storage an operating system actually mounts. S3 is the default choice for anything that isn't a running instance's own disk: static assets, backups, data lake input, build artifacts. EBS is per-instance block storage that lives and dies (or persists) independently of the instance it's attached to. EFS is the odd one out — an actual NFS-based shared filesystem multiple instances can mount concurrently, the tool to reach for when several EC2 instances or ECS tasks genuinely need to read and write the same files at once, which neither S3 nor EBS is built for (S3 has no real file-locking/POSIX semantics, and an EBS volume can only attach to multiple instances at all under the narrower Multi-Attach mode, and even then only in specific configurations).

The aws s3 vs. aws s3api split#

Every aws s3 command is a convenience wrapper Amazon built on top of the real API — it's not that aws s3 is "old" and s3api is "new," they serve different jobs. Reach for aws s3 for everyday object movement; reach for aws s3api the moment you need bucket configuration (versioning, policies, lifecycle rules, encryption), precise pagination/--query control, or any single-object metadata operation that has no aws s3 equivalent at all.

S3 — copying, syncing, and listing objects#

aws s3 cp file.txt s3://my-bucket/path/file.txt
aws s3 cp s3://my-bucket/path/file.txt ./file.txt
aws s3 sync ./local-dir s3://my-bucket/path/ --delete           # mirror local -> bucket, remove extras in destination
aws s3 sync s3://my-bucket/path/ ./local-dir
aws s3 sync ./local-dir s3://my-bucket/path/ --exclude "*.log" --include "important.log"
aws s3 ls s3://my-bucket/path/ --recursive --human-readable --summarize

--dryrun on cp/sync/rm previews what would change without doing it — always worth running once before a sync --delete against anything you can't easily rebuild. --exclude/--include are applied in order left to right, with later patterns able to re-include something an earlier --exclude already ruled out (as in the example above) — a common point of confusion for anyone expecting first-match-wins behavior.

S3 — removing objects and buckets#

aws s3 rm s3://my-bucket/path/file.txt
aws s3 rm s3://my-bucket/path/ --recursive               # delete everything under a prefix
aws s3 rb s3://my-bucket --force                          # delete a bucket, --force empties it first

S3 — bucket configuration (s3api)#

aws s3api create-bucket --bucket my-new-bucket --create-bucket-configuration LocationConstraint=us-west-2
aws s3api put-bucket-versioning --bucket my-bucket --versioning-configuration Status=Enabled
aws s3api put-bucket-policy --bucket my-bucket --policy file://bucket-policy.json
aws s3api put-bucket-encryption --bucket my-bucket --server-side-encryption-configuration '{
  "Rules": [{"ApplyServerSideEncryptionByDefault": {"SSEAlgorithm": "aws:kms"}, "BucketKeyEnabled": true}]
}'
aws s3api put-public-access-block --bucket my-bucket --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
aws s3api head-object --bucket my-bucket --key path/file.txt   # metadata only, no download

S3 — listing with s3api for scripting#

aws s3api list-objects-v2 --bucket my-bucket --prefix path/ --query 'Contents[].Key' --output text

s3api list-objects-v2 is the scriptable equivalent of aws s3 ls — reach for it over ls when you need --query filtering, precise pagination control (--max-items, --starting-token), or output piped directly into another AWS CLI call without parsing aws s3 ls's human-oriented text format.

S3 — versioning and listing versions#

aws s3api get-bucket-versioning --bucket my-bucket
aws s3api list-object-versions --bucket my-bucket --prefix path/file.txt
aws s3api get-object --bucket my-bucket --key path/file.txt --version-id <version-id> downloaded-old-version.txt

Once versioning is enabled on a bucket it cannot be disabled, only suspended (Status=Suspended) — plan for the storage cost before turning it on. A "deleted" object in a versioned bucket isn't gone; list-object-versions shows it with a delete-marker entry, and it's recoverable by fetching a specific --version-id, which is the standard recovery path for an accidental aws s3 rm against a versioned bucket.

S3 — lifecycle rules#

aws s3api put-bucket-lifecycle-configuration --bucket my-bucket --lifecycle-configuration file://lifecycle.json
aws s3api get-bucket-lifecycle-configuration --bucket my-bucket

A minimal lifecycle.json transitioning old objects to cheaper storage and expiring them later:

{
  "Rules": [
    {
      "ID": "archive-and-expire",
      "Filter": { "Prefix": "logs/" },
      "Status": "Enabled",
      "Transitions": [{ "Days": 30, "StorageClass": "GLACIER" }],
      "Expiration": { "Days": 365 }
    }
  ]
}

S3 — restoring an archived (Glacier) object#

aws s3api restore-object --bucket my-bucket --key path/file.txt \
  --restore-request '{"Days":7,"GlacierJobParameters":{"Tier":"Standard"}}'
aws s3api head-object --bucket my-bucket --key path/file.txt --query 'Restore'   # check restore status/progress

An object transitioned to Glacier/Deep Archive by a lifecycle rule isn't immediately downloadable — restore-object requests a temporary copy be rehydrated back to a retrievable state, which can take minutes (Expedited tier) to many hours (Bulk tier for Deep Archive), and --days controls how long the restored copy stays available before it's automatically re-archived. Attempting a plain aws s3 cp against a still-archived object fails outright rather than silently waiting.

S3 — cross-region/same-region replication#

aws s3api put-bucket-replication --bucket my-bucket --replication-configuration file://replication.json

Replication requires versioning enabled on both the source and destination buckets, and an IAM role (specified inside replication.json's Role field) with permission to read the source and write the destination. It's forward-only, not a one-time backfill — it silently does nothing for objects that existed before the rule was created.

S3 — presigned URLs#

aws s3 presign s3://my-bucket/path/file.txt --expires-in 3600

A presigned URL grants temporary access (default 1 hour, max 7 days for IAM-user/role credentials) to a single object using the credentials of whoever ran the command — anyone holding the URL can perform that action without their own AWS credentials, so treat the URL itself as a secret. It only works for actions the signing principal is actually allowed to perform (GetObject by default).

EBS — creating and attaching volumes#

aws ec2 create-volume --availability-zone us-east-1a --size 100 --volume-type gp3
aws ec2 attach-volume --volume-id vol-0123456789abcdef0 --instance-id i-0123456789abcdef0 --device /dev/xvdf
aws ec2 describe-volumes --volume-ids vol-0123456789abcdef0
aws ec2 delete-volume --volume-id vol-0123456789abcdef0

A volume must be in the same Availability Zone as the instance it attaches to — a common source of InvalidVolume.ZoneMismatch errors when scripting instance + volume creation together, especially when the instance's own AZ was chosen implicitly by whatever subnet it landed in.

EBS — resizing a volume without downtime#

aws ec2 modify-volume --volume-id vol-0123456789abcdef0 --size 200
aws ec2 describe-volumes-modifications --volume-ids vol-0123456789abcdef0 --query 'VolumesModifications[].[ModificationState,Progress]'

modify-volume can grow (never shrink) volume size, type, or IOPS on a currently attached, in-use volume on modern instance types — no detach, no downtime. describe-volumes-modifications is how you track that the resize actually finished; the CLI call returning success only means the request was accepted, not that the new capacity is live. After the modification completes, the OS still needs its own step to extend the filesystem onto the new space (growpart + resize2fs/xfs_growfs on Linux) — modify-volume never does that part.

EBS — snapshots#

aws ec2 create-snapshot --volume-id vol-0123456789abcdef0 --description "pre-migration backup"
aws ec2 describe-snapshots --owner-ids self
aws ec2 describe-snapshots --filters "Name=volume-id,Values=vol-0123456789abcdef0"
aws ec2 create-snapshots --instance-specification InstanceId=i-0123456789abcdef0 --description "crash-consistent full-instance backup"

A single create-snapshot snapshots one volume; create-snapshots (plural) snapshots every volume attached to an instance at once, guaranteeing they're crash-consistent with each other (all captured at the same instant relative to the running instance) — the right tool whenever an instance has multiple volumes that need to be restorable together, like a data volume and a separate log volume for the same database.

EFS — file systems and mount targets#

aws efs create-file-system --performance-mode generalPurpose --throughput-mode bursting --encrypted
aws efs describe-file-systems
aws efs create-mount-target --file-system-id fs-0123456789abcdef0 --subnet-id subnet-0123456789abcdef0 --security-groups sg-0123456789abcdef0
aws efs describe-mount-targets --file-system-id fs-0123456789abcdef0

EFS needs one mount target per Availability Zone you want to mount from — an EC2 instance in a subnet with no mount target for that file system will time out trying to mount it, not fail with a clear error, which makes this a slow, confusing failure to diagnose the first time you hit it.

EFS — lifecycle management (cost control)#

aws efs put-lifecycle-configuration --file-system-id fs-0123456789abcdef0 --lifecycle-policies \
  '[{"TransitionToIA":"AFTER_30_DAYS"},{"TransitionToPrimaryStorageClass":"AFTER_1_ACCESS"}]'

Like S3, EFS supports moving cold data to a cheaper Infrequent Access storage class automatically — TransitionToIA after N days of no access, and TransitionToPrimaryStorageClass moves a file back to Standard the moment it's accessed again, which matters because IA storage carries a per-request read charge that can make frequently-touched files more expensive there than on Standard, not less.

Real-world scenario: safely rotating S3 lifecycle policy on a bucket already holding years of data#

A team wants to add a 90-day Glacier transition to a bucket that's been accumulating logs for three years with no lifecycle rule at all:

# 1. See what's actually there before changing anything — total size and object count by age
aws s3api list-objects-v2 --bucket my-logs-bucket --prefix logs/2023/ --query 'length(Contents)'

# 2. Confirm no rule exists yet (an overwrite would silently replace it, not merge)
aws s3api get-bucket-lifecycle-configuration --bucket my-logs-bucket

# 3. Apply, scoped to a specific prefix first rather than the whole bucket
aws s3api put-bucket-lifecycle-configuration --bucket my-logs-bucket --lifecycle-configuration file://lifecycle.json

Real-world scenario: recovering an accidentally deleted object from a versioned bucket#

aws s3api list-object-versions --bucket my-bucket --prefix path/file.txt \
  --query 'DeleteMarkers[?Key==`path/file.txt`].[VersionId,LastModified]'
# Find the delete marker's VersionId, then remove *it* — this un-deletes the object
aws s3api delete-object --bucket my-bucket --key path/file.txt --version-id <delete-marker-version-id>

Deleting a specific object version that happens to be a delete marker restores the object to its previous state — a delete marker is itself just another version, and removing it is how you undo an aws s3 rm against a versioned bucket without needing a separate backup at all.

Real-world scenario: migrating a large dataset between buckets across accounts#

aws s3 sync s3://source-account-bucket/dataset/ s3://dest-account-bucket/dataset/ \
  --source-region us-east-1 --profile dest-account-profile

Cross-account sync needs the destination bucket's policy to explicitly grant the source principal s3:GetObject/s3:ListBucket, and (very commonly missed) the objects arriving in the destination bucket are owned by the account that performed the copy, not the destination bucket owner, unless the destination bucket has S3 Object Ownership set to BucketOwnerEnforced — otherwise the destination account can end up unable to manage objects it "owns."

CI/CD recipe: syncing a static site build to S3 and invalidating CloudFront#

# .github/workflows/deploy.yml
name: Deploy static site
on:
  push:
    branches: [main]
jobs:
  deploy:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: actions/checkout@v4
      - run: npm ci && npm run build
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/GitHubActionsDeployRole
          aws-region: us-east-1
      - run: aws s3 sync ./out s3://my-site-bucket/ --delete
      - run: aws cloudfront create-invalidation --distribution-id E1234567890 --paths "/*"

The invalidation step matters as much as the sync itself for a CDN-fronted static site — without it, CloudFront keeps serving the previous build from cache and the sync can look like it silently "didn't deploy," which is a real, recurring incident-shaped surprise for anyone new to a CDN-fronted pipeline.

Common pitfalls#

• aws s3 rm --recursive with no --dryrun first — see the caution above; there's no undo without versioning already having been on.
• put-bucket-lifecycle-configuration/put-bucket-replication silently replacing, not merging — see the IMPORTANT callout above; always fetch current config before writing a new one.
• Assuming modify-volume immediately delivers new capacity — the API call returning success only means the modification was accepted; poll describe-volumes-modifications, and remember the OS still needs its own growpart/resize2fs step afterward.
• Mounting an EFS file system from a subnet with no mount target in that AZ — this hangs (times out) rather than failing cleanly; check describe-mount-targets covers every AZ you actually need before debugging the client side.
• Trying to cp/get-object a Glacier-archived object without restoring it first — it fails immediately rather than waiting; restore-object (and checking head-object --query 'Restore') is a required step first.

Exit codes / when to reach for something else#

aws s3/s3api commands return 0 on success, non-zero on any failure — including a partial sync failure part-way through a large transfer, so check exit status rather than assuming a long-running sync that printed progress lines necessarily finished cleanly. For infrastructure-shaped storage config (bucket policies, lifecycle rules, EFS file systems meant to be reviewable and version controlled) prefer Terraform/CloudFormation over hand-run s3api put-* calls; reach for the CLI directly for the data-movement operations (cp, sync, restores, one-off migrations) that Terraform was never meant to manage in the first place.