Networking: VPC, Route 53 & ELB

What it is and where it fits#

Networking is the layer everything else in this series sits on top of — an EC2 instance, an ECS task, an EKS pod, an RDS database, all live inside a VPC's subnets and are reachable (or not) according to route tables, security groups, and NACLs configured here. This page covers VPC construction (subnets, routing, peering, transit gateways, PrivateLink endpoints), Route 53 for DNS, and load balancing via elbv2 (the modern Application/Network Load Balancer API — elb, singular, without the v2 suffix, is the older Classic Load Balancer API and shouldn't be reached for in new work). Nothing here is "declarative" the way Terraform is: every command below is one imperative API call, useful for building understanding, debugging connectivity, and scripting genuinely one-off network changes — production VPC topology is a strong candidate for staying in Terraform once it stabilizes.

VPCs and subnets#

aws ec2 create-vpc --cidr-block 10.0.0.0/16 --tag-specifications 'ResourceType=vpc,Tags=[{Key=Name,Value=my-vpc}]'
aws ec2 describe-vpcs --filters "Name=tag:Name,Values=my-vpc"
aws ec2 create-subnet --vpc-id vpc-0123456789abcdef0 --cidr-block 10.0.1.0/24 --availability-zone us-east-1a
aws ec2 describe-subnets --filters "Name=vpc-id,Values=vpc-0123456789abcdef0"

How a subnet becomes "public" — the whole mechanism#

aws ec2 create-internet-gateway
aws ec2 attach-internet-gateway --internet-gateway-id igw-0123456789abcdef0 --vpc-id vpc-0123456789abcdef0
aws ec2 create-route-table --vpc-id vpc-0123456789abcdef0
aws ec2 create-route --route-table-id rtb-0123456789abcdef0 --destination-cidr-block 0.0.0.0/0 --gateway-id igw-0123456789abcdef0
aws ec2 associate-route-table --route-table-id rtb-0123456789abcdef0 --subnet-id subnet-0123456789abcdef0

A subnet is only "public" because its associated route table has a 0.0.0.0/0 route pointing at an internet gateway — there is no separate "make this subnet public" flag anywhere in the API. This four-command chain (gateway → attach → route → associate) is the entire mechanism, and it's worth internalizing precisely because AWS consoles/tutorials often present "public subnet" as if it were an intrinsic subnet property rather than a route table's contents.

Private subnets and NAT gateways#

aws ec2 create-nat-gateway --subnet-id subnet-0123456789abcdef0 --allocation-id eipalloc-0123456789abcdef0
aws ec2 describe-nat-gateways --filter "Name=vpc-id,Values=vpc-0123456789abcdef0"
aws ec2 create-route --route-table-id rtb-0987654321fedcba0 --destination-cidr-block 0.0.0.0/0 --nat-gateway-id nat-0123456789abcdef0

A NAT gateway lives in a public subnet (it needs its own route to an internet gateway) but serves outbound-only internet access for instances in a private subnet whose route table points 0.0.0.0/0 at the NAT gateway instead of the internet gateway directly. This gives private-subnet instances a way to reach the internet (for package updates, external API calls) without being reachable from the internet — the NAT gateway only forwards traffic that originated from inside the VPC.

Security group and NACL rules#

aws ec2 authorize-security-group-ingress --group-id sg-0123456789abcdef0 --protocol tcp --port 443 --cidr 0.0.0.0/0
aws ec2 authorize-security-group-ingress --group-id sg-0123456789abcdef0 --protocol tcp --port 22 --source-group sg-0123456789abcdef1
aws ec2 create-network-acl-entry --network-acl-id acl-0123456789abcdef0 --rule-number 100 \
  --protocol tcp --port-range From=443,To=443 --cidr-block 0.0.0.0/0 --rule-action allow --ingress

Security groups are stateful (a response to an allowed inbound request is automatically allowed back out, no matching egress rule needed) and evaluate every rule (allow-only, implicit deny) — network ACLs are stateless (return traffic needs its own explicit rule in the opposite direction) and evaluate rules in numeric order, first match wins, supporting explicit deny. Most day-to-day access control lives in security groups; NACLs are typically reserved for a coarse, subnet-wide explicit deny (blocking a known-bad CIDR range at the subnet boundary) rather than fine-grained per-resource rules.

VPC peering#

aws ec2 create-vpc-peering-connection --vpc-id vpc-0123456789abcdef0 --peer-vpc-id vpc-0fedcba9876543210
aws ec2 accept-vpc-peering-connection --vpc-peering-connection-id pcx-0123456789abcdef0    # run in the peer account/region
aws ec2 describe-vpc-peering-connections --filters "Name=status-code,Values=active"

Creating the connection only proposes it — it stays pending-acceptance until the owner of the peer VPC runs accept-vpc-peering-connection. Peering also doesn't add routes automatically: both VPCs' route tables still need an explicit create-route pointing at the pcx- connection ID, same pattern as the internet gateway routing above.

Transit gateway#

aws ec2 create-transit-gateway --description "hub-vpc-tgw"
aws ec2 create-transit-gateway-vpc-attachment --transit-gateway-id tgw-0123456789abcdef0 --vpc-id vpc-0123456789abcdef0 --subnet-ids subnet-0123456789abcdef0
aws ec2 describe-transit-gateways --filters "Name=state,Values=available"

A transit gateway is the hub-and-spoke alternative to VPC peering's mesh — worth it once you're peering more than a handful of VPCs, since peering connections don't transit (VPC A peered to B and B peered to C does not let A reach C, but attaching A, B, and C to the same transit gateway does). The crossover point where a mesh of peering connections becomes harder to reason about than a transit gateway is typically somewhere around 4-5 VPCs — below that, peering's simplicity usually wins.

VPC endpoints (PrivateLink)#

aws ec2 create-vpc-endpoint --vpc-id vpc-0123456789abcdef0 --service-name com.amazonaws.us-east-1.s3 --route-table-ids rtb-0123456789abcdef0
aws ec2 create-vpc-endpoint --vpc-id vpc-0123456789abcdef0 --vpc-endpoint-type Interface --service-name com.amazonaws.us-east-1.ec2 --subnet-ids subnet-0123456789abcdef0 --security-group-ids sg-0123456789abcdef0
aws ec2 describe-vpc-endpoints --filters "Name=vpc-id,Values=vpc-0123456789abcdef0"

--vpc-endpoint-type defaults to Gateway (only S3 and DynamoDB support it — attaches to a route table, no hourly cost) versus Interface (an ENI-backed endpoint for most other services — billed hourly, needs subnets and a security group). Either way, traffic to that AWS service stays on the AWS network instead of routing out through a NAT gateway or the public internet — worth reaching for deliberately in a private-subnet architecture both for security posture and to cut NAT gateway data processing charges for high-volume AWS API traffic (S3 in particular).

Route 53 — hosted zones#

aws route53 list-hosted-zones
aws route53 create-hosted-zone --name example.com --caller-reference "$(date +%s)"

--caller-reference must be a unique string per call (it's an idempotency token, not a DNS field) — a timestamp or UUID both work; a fixed literal fails on a second run with the same name, which is a common surprise for anyone treating it as just another required parameter.

Route 53 — records#

aws route53 list-resource-record-sets --hosted-zone-id Z1234567890ABC
aws route53 change-resource-record-sets --hosted-zone-id Z1234567890ABC --change-batch '{
  "Changes": [{
    "Action": "UPSERT",
    "ResourceRecordSet": {
      "Name": "app.example.com",
      "Type": "A",
      "TTL": 300,
      "ResourceRecords": [{"Value": "203.0.113.10"}]
    }
  }]
}'

change-resource-record-sets is the only way to create/update/delete records — there's no create-record shortcut. UPSERT creates the record if it doesn't exist or overwrites it if it does; use CREATE/DELETE when you specifically want the API to reject a call that doesn't match the record's current existence state, a useful safety check in a script that shouldn't silently overwrite an unrelated existing record with the same name.

Route 53 — health checks and failover routing#

aws route53 create-health-check --caller-reference "$(date +%s)" --health-check-config \
  '{"IPAddress":"203.0.113.10","Port":443,"Type":"HTTPS","ResourcePath":"/healthz","RequestInterval":30,"FailureThreshold":3}'
aws route53 get-health-check-status --health-check-id abc12345-6789-def0-1234-56789abcdef0

A Route 53 health check is a separate, independently-billed resource from an ALB target-group health check — reference its ID in a record's HealthCheckId field to make Route 53 stop returning that record when the check fails, the DNS-layer building block for active-active or active-passive multi-region failover. Don't create Route 53 health checks for EC2 instances that are already behind an ELB, though — the load balancer's own health checks already cover that; a redundant Route 53 check against the same instances adds cost and confusion without adding real coverage.

Load balancers (ALB/NLB) — creation and listeners#

aws elbv2 create-load-balancer --name my-alb --subnets subnet-0123456789abcdef0 subnet-0123456789abcdef1 --security-groups sg-0123456789abcdef0 --type application
aws elbv2 describe-load-balancers --names my-alb
aws elbv2 create-target-group --name my-targets --protocol HTTP --port 80 --vpc-id vpc-0123456789abcdef0 --health-check-path /healthz
aws elbv2 create-listener --load-balancer-arn <lb-arn> --protocol HTTP --port 80 --default-actions Type=forward,TargetGroupArn=<target-group-arn>

--type application (ALB, Layer 7 — path/host-based routing, needs at least two subnets in different AZs) vs. --type network (NLB, Layer 4 — raw TCP/UDP, extreme throughput, a static IP per AZ). Choose NLB for a workload that needs a fixed IP (VPN endpoints, some legacy integrations) or extreme performance; ALB for anything HTTP(S)-shaped that benefits from path-based routing or WAF integration.

Modifying a listener — adding HTTPS, changing routing rules#

aws elbv2 modify-listener --listener-arn <listener-arn> --protocol HTTPS --port 443 \
  --certificates CertificateArn=<acm-cert-arn> --ssl-policy ELBSecurityPolicy-TLS13-1-2-2021-06
aws elbv2 create-rule --listener-arn <listener-arn> --priority 10 \
  --conditions Field=path-pattern,Values='/api/*' --actions Type=forward,TargetGroupArn=<api-target-group-arn>

Changing a listener's protocol from HTTPS to HTTP (or TLS to TCP) drops the security policy and certificate — going the other direction requires supplying both again, they're never auto-populated. create-rule lets one ALB listener route different paths/hosts to different target groups — the mechanism behind running several services behind one load balancer instead of provisioning one ALB per service.

Target groups — registering and checking health#

aws elbv2 register-targets --target-group-arn <target-group-arn> --targets Id=i-0123456789abcdef0
aws elbv2 describe-target-health --target-group-arn <target-group-arn>

describe-target-health is the fastest way to confirm whether an ALB actually considers your instances/tasks healthy — a TargetHealth.State of unhealthy here, not application logs, is usually the first place to look when a load balancer is returning 502s/503s. A TargetHealth.Reason of Target.Timeout typically means the health-check path is slow or unresponsive; Target.FailedHealth Checks means it's actively returning a non-2xx status — different root causes worth distinguishing before digging into application logs.

Real-world scenario: diagnosing "the ALB returns 503s but the app looks fine"#

# 1. Are there even any healthy targets registered right now?
aws elbv2 describe-target-health --target-group-arn <target-group-arn> --query 'TargetHealthDescriptions[].[Target.Id,TargetHealth.State,TargetHealth.Reason]'

# 2. Does the security group attached to the targets actually allow the ALB's security group in
#    on the health-check port?
aws ec2 describe-security-groups --group-ids <target-sg-id> --query 'SecurityGroups[].IpPermissions'

An ALB returning 503 with no healthy targets is nearly always one of two things: either the application genuinely isn't responding on the health-check path/port, or — the one that looks like "the app is fine" from inside the instance — the target's security group doesn't allow inbound traffic from the ALB's security group on the health-check port at all, so the check never even reaches the application to fail cleanly.

Real-world scenario: locking a security group down from 0.0.0.0/0 after an audit finding#

# 1. See exactly what's open first
aws ec2 describe-security-groups --group-ids sg-0123456789abcdef0 --query 'SecurityGroups[].IpPermissions'

# 2. Add the scoped replacement rule before removing the broad one — never the other way around
aws ec2 authorize-security-group-ingress --group-id sg-0123456789abcdef0 --protocol tcp --port 443 --cidr 203.0.113.0/24
aws ec2 revoke-security-group-ingress --group-id sg-0123456789abcdef0 --protocol tcp --port 443 --cidr 0.0.0.0/0

CI/CD recipe: validating a Terraform-managed VPC change against real routing before merge#

# .github/workflows/vpc-drift-check.yml
name: VPC drift check
on: [pull_request]
jobs:
  drift-check:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/ReadOnlyCIRole
          aws-region: us-east-1
      - name: Confirm no unexpected 0.0.0.0/0 ingress exists on production security groups
        run: |
          OPEN=$(aws ec2 describe-security-groups \
            --filters "Name=tag:Environment,Values=production" \
            --query "SecurityGroups[].IpPermissions[?contains(IpRanges[].CidrIp, '0.0.0.0/0')]" \
            --output text)
          if [ -n "$OPEN" ]; then echo "Unexpected open ingress found"; exit 1; fi

A read-only CI job like this is a cheap, high-value guardrail independent of whatever Terraform/policy tooling already gates the actual infrastructure change — it queries live AWS state directly, so it still catches a manual console change that bypassed Terraform entirely.

Common pitfalls#

• Believing a subnet is "public" as an intrinsic property — it's entirely a function of its route table's contents; moving a subnet between route tables can silently make it public or private.
• Forgetting peering/transit-gateway connections don't add routes automatically — the connection existing and being active is necessary but not sufficient; both sides still need explicit routes.
• Revoking a broad security group rule before the replacement narrower rule is in place — see the WARNING above; always add-then-revoke, never revoke-then-add.
• Creating a Route 53 health check for instances already behind an ELB — redundant with the load balancer's own health checks; added cost with no added coverage.
• Not checking the target's security group when an ALB reports "no healthy targets" — the application can be completely healthy while the health check still never reaches it.

Exit codes / when to reach for something else#

Exit 0 means the API accepted the request — for a routing or security-group change, that's not the same as "traffic is now flowing as intended," which is why the diagnostic scenarios above lean on describe-*/--query calls, not just the mutating call's own exit status. Prefer Terraform for VPC topology and security group rules meant to be reviewable, version-controlled, and consistent across environments; reach for these commands directly for live debugging (describe-target-health, describe-security-groups), emergency response (the security-group lockdown scenario above), and verifying that infrastructure someone else provisioned is actually configured the way it's supposed to be.