Observability: CloudWatch & Logs

What it is and where it fits#

CloudWatch is AWS's native observability service, split into two halves the CLI's own command structure mirrors: aws cloudwatch for numeric time-series data (metrics, alarms, dashboards) and aws logs for text log data (log groups, streams, and the CloudWatch Logs Insights query language). Every AWS-managed service ships metrics here automatically at no extra setup cost, which makes CloudWatch the default first place to look when something's wrong in AWS-native infrastructure — even teams running Prometheus/Grafana or a third-party observability stack elsewhere typically still end up reading CloudWatch for anything AWS emits natively (RDS, ELB, Lambda concurrency, EBS IOPS) rather than re-instrumenting it. This page covers reading and publishing metrics, alarming on them, and querying logs — the everyday loop of "is something wrong, and if so, what does the evidence say."

How metrics, alarms, and logs relate#

A metric filter (not covered as a dedicated section below, but worth knowing exists) can turn a log pattern into a metric — e.g. counting "ERROR" occurrences per minute in application logs — which lets you alarm on log content using the same put-metric-alarm mechanism as any native AWS metric, bridging the two halves of CloudWatch shown above.

Reading metrics#

aws cloudwatch get-metric-statistics \
  --namespace AWS/EC2 --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --start-time 2026-08-19T00:00:00Z --end-time 2026-08-20T00:00:00Z \
  --period 300 --statistics Average

aws cloudwatch get-metric-data \
  --metric-data-queries '[{"Id":"cpu","MetricStat":{"Metric":{"Namespace":"AWS/EC2","MetricName":"CPUUtilization","Dimensions":[{"Name":"InstanceId","Value":"i-0123456789abcdef0"}]},"Period":300,"Stat":"Average"},"ReturnData":true}]' \
  --start-time 2026-08-19T00:00:00Z --end-time 2026-08-20T00:00:00Z

get-metric-statistics is the simple single-metric query; get-metric-data is the newer, batched form (query up to 500 metrics in one call, supports metric math — see below) with generally lower latency per metric returned. Reach for get-metric-data for anything beyond a quick one-off check; it's also the form the console's own dashboards use internally.

Publishing custom metrics#

aws cloudwatch put-metric-data \
  --namespace MyApp \
  --metric-name QueueDepth \
  --value 42 --unit Count \
  --dimensions Environment=production

--namespace is the top-level grouping (never starts with AWS/, which is reserved for AWS's own service metrics) — pick one per application or logical system, since it's also the unit of cost/isolation in the CloudWatch console's namespace browser. A common mistake is publishing every metric under one generic namespace shared across unrelated apps, which makes the metric browser useless once a team has more than a handful of services.

Alarms#

aws cloudwatch put-metric-alarm \
  --alarm-name high-cpu \
  --namespace AWS/EC2 --metric-name CPUUtilization \
  --dimensions Name=InstanceId,Value=i-0123456789abcdef0 \
  --statistic Average --period 300 --evaluation-periods 3 \
  --threshold 80 --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:my-alerts-topic

aws cloudwatch describe-alarms --state-value ALARM       # only alarms currently firing
aws cloudwatch describe-alarms --alarm-name-prefix high- # filter by name prefix

--evaluation-periods 3 with --period 300 means the alarm needs 3 consecutive 5-minute periods above threshold before it actually fires (15 minutes of sustained breach) — a common first-time misconfiguration is setting --evaluation-periods 1, which makes an alarm trigger on a single noisy spike instead of a sustained condition, generating pager fatigue almost immediately.

Testing an alarm without waiting for a real breach#

aws cloudwatch set-alarm-state --alarm-name high-cpu --state-value ALARM --state-reason "Testing notification pipeline"
aws cloudwatch describe-alarm-history --alarm-name high-cpu --history-item-type StateUpdate

set-alarm-state temporarily forces an alarm's state for testing — it does trigger the configured actions (an SNS message really does go out), which makes it the right tool for verifying an on-call notification pipeline actually works end to end, not just that the alarm's threshold math is correct. The alarm returns to its real, metric-derived state on the next evaluation, typically within seconds to minutes depending on --period.

Metric math#

aws cloudwatch get-metric-data \
  --start-time 2026-08-20T00:00:00Z --end-time 2026-08-21T00:00:00Z \
  --metric-data-queries '[
    {"Id":"m1","MetricStat":{"Metric":{"Namespace":"AWS/EBS","MetricName":"VolumeReadOps","Dimensions":[{"Name":"VolumeId","Value":"vol-0123456789abcdef0"}]},"Period":300,"Stat":"Sum"},"ReturnData":false},
    {"Id":"m2","MetricStat":{"Metric":{"Namespace":"AWS/EBS","MetricName":"VolumeWriteOps","Dimensions":[{"Name":"VolumeId","Value":"vol-0123456789abcdef0"}]},"Period":300,"Stat":"Sum"},"ReturnData":false},
    {"Id":"total_iops","Expression":"(m1+m2)/300","Label":"Avg Total IOPS","ReturnData":true}
  ]'

Metric math lets you combine raw metrics with arithmetic/statistical functions server-side instead of pulling raw series and computing client-side. ReturnData: false on the input metrics (m1, m2) hides them from the response and returns only the computed expression — set it true on any series you also want returned alongside the math result, useful when you want both the raw inputs and the derived value plotted together.

Composite alarms#

aws cloudwatch put-composite-alarm \
  --alarm-name service-degraded \
  --alarm-rule "ALARM(high-cpu) AND ALARM(high-error-rate)" \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:my-alerts-topic \
  --actions-enabled

A composite alarm doesn't watch a metric directly — its --alarm-rule is a boolean expression (AND/OR/NOT, parenthesized) over the ALARM/OK/INSUFFICIENT_DATA state of other alarms. Use it to cut noise: page only when several individually-noisy alarms are in ALARM together, instead of firing one page per underlying alarm — the standard fix for an on-call rotation that's drowning in correlated, redundant pages for what is really one underlying incident.

Dashboards#

aws cloudwatch put-dashboard --dashboard-name my-service --dashboard-body file://dashboard.json
aws cloudwatch get-dashboard --dashboard-name my-service
aws cloudwatch list-dashboards --dashboard-name-prefix my-

--dashboard-body is a JSON document of widget definitions (each widget references a metric or a Logs Insights query) — there's no imperative "add a widget" command; put-dashboard always replaces the entire dashboard body. Fetch the current body with get-dashboard, edit it, and put-dashboard it back rather than hand-authoring the whole thing from scratch each time, and treat the JSON as something worth keeping in version control alongside the rest of the service's infrastructure.

Log groups and streams#

aws logs describe-log-groups --log-group-name-prefix /aws/lambda/
aws logs describe-log-streams --log-group-name /aws/lambda/my-function --order-by LastEventTime --descending
aws logs put-retention-policy --log-group-name /aws/lambda/my-function --retention-in-days 30

Log groups default to never expiring unless you set a retention policy — a common, quietly expensive default left over from first-time Lambda/ECS setups, since CloudWatch Logs storage cost scales with retained volume indefinitely by default. Worth auditing account-wide with describe-log-groups periodically, and worth setting put-retention-policy explicitly as part of provisioning any new log group from day one rather than as a later cleanup task.

Reading log events#

aws logs get-log-events --log-group-name /aws/lambda/my-function --log-stream-name <stream-name> --start-from-head
aws logs filter-log-events --log-group-name /aws/lambda/my-function --filter-pattern "ERROR" --start-time 1755648000000
aws logs tail /aws/lambda/my-function --follow --since 1h --filter-pattern "ERROR"

aws logs tail is the closest thing to kubectl logs -f for CloudWatch — it's a CLI-only convenience command (not a direct API wrapper), it accepts human-readable --since values like 1h/30m, and --follow streams new events live instead of returning a fixed page. filter-log-events searches across every stream in a log group at once; get-log-events reads one specific stream. Use filter-log-events (or logs tail) when you don't already know which stream/task/container instance produced the log line you're looking for, which in practice is most incident-investigation situations.

CloudWatch Logs Insights — querying across log groups#

aws logs start-query \
  --log-group-names /aws/lambda/my-function \
  --start-time $(date -d '1 hour ago' +%s) --end-time $(date +%s) \
  --query-string 'fields @timestamp, @message | filter @message like /ERROR/ | sort @timestamp desc | limit 50'

aws logs get-query-results --query-id abc12345-6789-def0-1234-56789abcdef0

start-query and get-query-results are two separate calls because Insights queries run asynchronously — start-query returns a queryId immediately, and the query itself may still be running when get-query-results is first called (check the status field in the response; poll again if it's Running, not Complete). This two-step, poll-based shape is the same async pattern the credential report in the IAM page uses, and shows up across several AWS APIs whenever an operation takes longer than a typical synchronous request-response cycle. A single query can span up to 10 log groups at once — the tool of choice for "search across every Lambda function's logs for this request ID" rather than checking each function's log group individually.

Real-world scenario: finding the one bad deploy in a fleet of instances#

An alarm fires for elevated error rate across a service running on 20 EC2 instances behind an ALB, and CloudWatch alone can't say which instance(s) are actually unhealthy without checking each one:

# 1. Confirm the alarm and get its evaluation window
aws cloudwatch describe-alarms --alarm-names high-error-rate --query 'MetricAlarms[0].[StateReason,StateUpdatedTimestamp]'

# 2. Search application logs across every instance's log stream at once for the error signature
aws logs filter-log-events --log-group-name /app/checkout-service \
  --start-time $(date -d '20 minutes ago' +%s000) --filter-pattern "\"500 Internal Server Error\""

# 3. Cross-reference which instance IDs appear in the matching log streams
aws logs filter-log-events --log-group-name /app/checkout-service --filter-pattern "500" \
  --query 'events[].logStreamName' --output text | sort -u

filter-log-events' logStreamName field is often the fastest link between "an error happened" and "on which specific host" when the log stream naming convention includes the instance/task ID (a worthwhile convention to establish deliberately when configuring the log driver, precisely so this kind of query works later).

Real-world scenario: reducing alarm noise for a genuinely bursty metric#

A queue-depth metric legitimately spikes for a few minutes during a nightly batch job, triggering a false alarm every night:

aws cloudwatch put-metric-alarm \
  --alarm-name queue-depth-sustained \
  --namespace MyApp --metric-name QueueDepth \
  --statistic Average --period 300 --evaluation-periods 6 \
  --threshold 1000 --comparison-operator GreaterThanThreshold \
  --datapoints-to-alarm 4 \
  --treat-missing-data notBreaching

Real-world scenario: setting up log-based alerting without a separate agent#

An application logs structured JSON but has no CloudWatch agent configured to extract a metric from it:

aws logs put-metric-filter --log-group-name /app/checkout-service \
  --filter-name payment-failures --filter-pattern '{ $.level = "ERROR" && $.event = "payment_failed" }' \
  --metric-transformations metricName=PaymentFailures,metricNamespace=MyApp,metricValue=1,defaultValue=0

aws cloudwatch put-metric-alarm --alarm-name payment-failure-spike \
  --namespace MyApp --metric-name PaymentFailures --statistic Sum --period 60 \
  --evaluation-periods 5 --threshold 10 --comparison-operator GreaterThanThreshold \
  --alarm-actions arn:aws:sns:us-east-1:111122223333:my-alerts-topic

A metric filter turns a log pattern match into an emitted data point every time it matches — no application code change and no separate metrics-emission library needed, since the same log lines the application already writes drive the alarm. defaultValue=0 matters: without it, the metric simply has no data point in periods with zero matching log lines (instead of an explicit 0), which can make --evaluation-periods/--treat-missing-data behave unexpectedly around quiet periods.

CI/CD recipe: gating a deploy on post-deploy error rate#

# .github/workflows/post-deploy-check.yml
name: Post-deploy health gate
on:
  workflow_run:
    workflows: ["Deploy"]
    types: [completed]
jobs:
  health-check:
    runs-on: ubuntu-latest
    permissions:
      id-token: write
      contents: read
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/GitHubActionsReadOnlyRole
          aws-region: us-east-1
      - name: Wait 5 minutes then check error rate
        run: |
          sleep 300
          ERRORS=$(aws cloudwatch get-metric-statistics \
            --namespace MyApp --metric-name Errors --statistic Sum \
            --start-time "$(date -u -d '5 minutes ago' +%Y-%m-%dT%H:%M:%S)" \
            --end-time "$(date -u +%Y-%m-%dT%H:%M:%S)" --period 300 \
            --query 'Datapoints[0].Sum' --output text)
          if [ "$ERRORS" != "None" ] && [ "$(echo "$ERRORS > 50" | bc)" -eq 1 ]; then
            echo "Error rate exceeded threshold post-deploy: $ERRORS"; exit 1
          fi

A post-deploy metric gate like this closes the loop a deploy pipeline otherwise leaves open — the aws ecs wait services-stable pattern from the compute page confirms tasks came up healthy at the infrastructure level, but only a real metric check confirms the new code isn't actively producing errors once it's serving real traffic.

Common pitfalls#

• --evaluation-periods 1 — triggers on a single noisy spike; see the Alarms section above for why most alarms should require sustained breach.
• Log groups with no retention policy set — accumulates cost indefinitely by default; audit with describe-log-groups and set put-retention-policy on every group.
• Forgetting get-query-results needs to be polled — start-query returns before the Insights query has actually finished; a get-query-results call made too early returns a Running status, not the final result, and a script that treats that as "no results" is wrong, not just slow.
• A metric filter with no defaultValue — leaves genuinely-zero periods with no data point at all instead of an explicit 0, which can distort --evaluation-periods/--treat-missing-data behavior.
• Reaching for describe-alarms --state-value ALARM as the only signal something's wrong — an alarm can be in INSUFFICIENT_DATA (no data flowing at all, often worse than a clean breach) without ever entering ALARM; check for that state too in any automated health check.

Exit codes / when to reach for something else#

CloudWatch/Logs commands exit 0 on a successfully accepted API call — for put-metric-alarm and put-metric-filter that means the configuration was saved, not that the alarm has actually evaluated yet (there's a real delay before the first evaluation completes). For alarm/dashboard definitions meant to be reviewable and consistent across environments, prefer Terraform/CloudFormation over hand-run put-metric-alarm/put-dashboard calls; reach for the CLI directly for ad-hoc investigation (filter-log-events, get-metric-data), scripted post-deploy health gates, and testing an alerting pipeline end to end with set-alarm-state. For a team already running its own metrics/log stack (Prometheus + Loki, Datadog, and similar), CloudWatch is still worth querying for anything AWS-managed emits natively that the third-party stack doesn't already scrape — it's rarely an either/or choice in practice.