Monitoring, Logging & Operations

Assumes you're comfortable with every prior chapter — this closing chapter is where you learn to observe and operate everything this course built. This site's SRE & Observability courses (this GCP series' Course 3, and the standalone SRE Fundamentals series) go far deeper into the discipline this chapter only introduces at ACE-exam depth.

Table of Contents#

1. What This Chapter Covers
2. Cloud Monitoring: Metrics, Dashboards, and Alerting Policies
3. Creating Custom and Log-Based Metrics
4. Cloud Logging: The Log Router and Log Buckets
5. The Logs Explorer and Logging Query Language
6. Exporting Logs to BigQuery, Cloud Storage, and Pub/Sub
7. Audit Logs: Admin Activity, Data Access, and System Event
8. The Ops Agent
9. Google Cloud Managed Service for Prometheus
10. Cloud Trace — Distributed Latency Analysis
11. Cloud Profiler — Continuous CPU and Heap Profiling
12. Error Reporting
13. Diagnostic Tools: Query Insights and the Index Advisor
14. Active Assist and Personalized Service Health
15. Gemini Cloud Assist for Monitoring and Log Analysis
16. A Full Worked Example: Instrumenting shipment-api End to End
17. Real-World Scenario: The Alert Nobody Trusted
18. Observability Terminology Map
19. How the Five Cloud Operations Products Fit Together
20. Pre-Flight Checklist: Is This Observability Setup Production-Ready?
21. Common Mistakes and Interview Traps
22. Worked Practice Problems
23. Summary and What's Next: Where This Course Goes From Here

What This Chapter Covers#

Every prior chapter built something; this chapter makes it observable. Cloud Monitoring and Cloud Logging are GCP's core operations products, joined by Cloud Trace, Cloud Profiler, Error Reporting, and Google Cloud Managed Service for Prometheus — together, GCP's five-product "Cloud Operations" suite, covering the ACE exam's "monitoring and logging" objective at the depth a working cloud engineer needs day to day.

🎯 By the end of this chapter, you'll be able to configure alerts that reach a human reliably, query logs efficiently instead of scrolling through them, and diagnose a real production issue using the right diagnostic tool for the actual symptom rather than guessing.

Cloud Monitoring: Metrics, Dashboards, and Alerting Policies#

Cloud Monitoring collects metrics from every GCP resource automatically — no configuration needed for the basics — and lets you build dashboards and alerting policies on top of them.

# An alerting policy on shipment-api's Cloud Run error rate —
# routed to the same on-call channel Part 2's budget alerts use
gcloud alpha monitoring policies create \
  --notification-channels=projects/meridian-shipment-prod/notificationChannels/CHANNEL_ID \
  --display-name="shipment-api elevated error rate" \
  --condition-display-name="Error rate above 5%" \
  --condition-filter='resource.type="cloud_run_revision" AND metric.type="run.googleapis.com/request_count" AND metric.label.response_code_class="5xx"' \
  --condition-threshold-value=0.05 \
  --condition-threshold-duration=300s

Creating Custom and Log-Based Metrics#

A log-based metric turns a pattern in your logs into a first-class Cloud Monitoring metric, chartable and alertable exactly like a native infrastructure metric.

# Count occurrences of a specific application-level error pattern
# in shipment-api's logs, as a metric Cloud Monitoring can alert on
gcloud logging metrics create shipment-api-payment-failures \
  --description="Count of payment processing failures" \
  --log-filter='resource.type="cloud_run_revision" AND resource.labels.service_name="shipment-api" AND jsonPayload.event="payment_failure"'

Log-based metrics can be scoped at the project level (visible everywhere) or the log-bucket level (scoped to logs already routed to that bucket) — the bucket-scoped form is genuinely useful once an aggregated sink (covered later in this chapter) is already routing multiple projects' logs into one central bucket, letting a metric apply consistently across all of them without redefining it per project.

One alerting policy can fan out to multiple notification channels simultaneously — Meridian routes every alert to both PagerDuty (for guaranteed human response) and a Slack channel (for team-wide visibility), matching this chapter's own advice to consolidate onto trusted paths rather than scattering alerts across disconnected surfaces.

Cloud Logging: The Log Router and Log Buckets#

Every log entry passes through the Log Router before landing anywhere — the mechanism that decides which log bucket(s) a given entry ends up in, and whether it's also exported elsewhere.

# A log sink routing audit logs from every project under the
# organization into Meridian's central security log bucket —
# the mechanism Part 1's landing zone bootstrap referenced without
# yet explaining
gcloud logging sinks create org-audit-sink \
  logging.googleapis.com/projects/meridian-shared-logging/locations/global/buckets/security-audit-logs \
  --organization=847213590482 --include-children \
  --log-filter='logName:"cloudaudit.googleapis.com"'

The Logs Explorer and Logging Query Language#

The Logging Query Language supports comparison operators, AND/OR combinations, and functions like timestamp() — genuinely closer to a real query language than a simple keyword search, worth learning properly rather than relying purely on the console's clickable filter builder for anything beyond a trivial lookup.

Exporting Logs to BigQuery, Cloud Storage, and Pub/Sub#

gcloud logging sinks create audit-logs-to-bigquery \
  bigquery.googleapis.com/projects/meridian-shared-logging/datasets/audit_logs \
  --log-filter='logName:"cloudaudit.googleapis.com"'

Audit Logs: Admin Activity, Data Access, and System Event#

# Enable Data Access logs for Cloud Storage specifically —
# deliberate, not blanket, given the volume/cost trade-off
gcloud projects get-iam-policy meridian-shipment-prod --format=json > policy.json
# (edit policy.json to add an auditConfigs block for storage.googleapis.com)
gcloud projects set-iam-policy meridian-shipment-prod policy.json

The Ops Agent#

# Deploy the Ops Agent — the unified successor to the older,
# separate Monitoring and Logging agents — via VM Manager's OS
# Config Management, the same mechanism Part 4 used for fleet-wide
# configuration enforcement
gcloud compute instances ops-agents policies create gps-worker-ops-agent \
  --project=meridian-shipment-prod \
  --zone=us-central1-a \
  --agent-rules=type=logging,version=latest,package-state=installed \
  --agent-rules=type=metrics,version=latest,package-state=installed \
  --group-labels=team=platform

This is the concrete implementation behind Part 4's bootstrap-gps-worker.sh startup script, which installed the Ops Agent as one of its first actions — every GPS-ingestion instance reports metrics and logs from the moment it boots, never as a manual follow-up step someone might forget.

Google Cloud Managed Service for Prometheus#

A fully managed, Prometheus-compatible metrics backend — the same PromQL query language and the same scrape-based collection model teams already know from self-hosted Prometheus, without operating the collection infrastructure.

# A PodMonitoring resource on GKE — scrapes route-optimizer's own
# /metrics endpoint using standard Prometheus conventions
apiVersion: monitoring.googleapis.com/v1
kind: PodMonitoring
metadata:
  name: route-optimizer-monitoring
  namespace: route-optimizer
spec:
  selector:
    matchLabels:
      app: route-optimizer
  endpoints:
    - port: metrics
      interval: 30s

Cloud Trace — Distributed Latency Analysis#

Cloud Trace shows exactly where time is spent across a request's full path through multiple services — the concrete tool behind the "correlate trace IDs with structured logs" skill referenced but not detailed in earlier chapters.

# Application-level tracing instrumentation via OpenTelemetry,
# GCP's currently-recommended path over the older Cloud Trace SDK
from opentelemetry import trace
tracer = trace.get_tracer("shipment-api")

with tracer.start_as_current_span("process_tracking_request"):
    with tracer.start_as_current_span("query_orders_db"):
        result = query_database(shipment_id)

A trace waterfall for a slow request shows exactly which span (the database query, an external API call, application logic) consumed the time — the difference between "the request was slow" and "the database query specifically took 800ms of a 900ms total," a genuinely different, more actionable diagnosis.

Cloud Profiler — Continuous CPU and Heap Profiling#

Cloud Profiler continuously samples a running application's CPU and memory usage in production, at low overhead — free, and genuinely unusual among observability tools for having no cost barrier to adoption.

import googlecloudprofiler
googlecloudprofiler.start(service="shipment-api", service_version="v13")

Where Cloud Trace shows where time goes across a request, Profiler shows where CPU/memory goes within the application's own code — a different, complementary question, useful for finding an inefficient function consuming disproportionate CPU even when no single request looks obviously slow in a trace.

Error Reporting#

Error Reporting automatically groups and surfaces application exceptions, free, with no separate configuration beyond having exceptions logged in a recognized format.

# Structured error logging that Error Reporting automatically
# detects and groups by stack trace similarity
import logging
try:
    process_shipment(shipment_id)
except Exception:
    logging.exception("Failed to process shipment")

Diagnostic Tools: Query Insights and the Index Advisor#

Query Insights (for Cloud SQL) surfaces the actual slow queries hitting a database instance, and the Index Advisor recommends specific indexes based on real observed query patterns — the concrete tools behind Part 6's database-performance discussions.

gcloud sql instances patch meridian-orders-db --insights-config-query-insights-enabled

Active Assist and Personalized Service Health#

Active Assist surfaces GCP-generated recommendations (idle resources, over-provisioned VMs, unused IAM permissions — the same IAM Recommender from Part 3 is one of its component insights) across cost, security, performance, and reliability. Personalized Service Health shows GCP incidents filtered to only the services and regions your own resources actually use, rather than a generic status page covering products you don't run at all.

Gemini Cloud Assist for Monitoring and Log Analysis#

Referenced across earlier chapters as an emerging AI-assisted layer, worth its concrete tie-in here: asking Gemini Cloud Assist "why did shipment-api's p99 latency spike at 3pm" correlates Monitoring metrics, Logging entries, and Trace data automatically — genuinely faster than manually cross-referencing three separate tools, though (per Part 2's own AI-tooling guidance) the output should be reviewed against the actual underlying data before being treated as the final diagnosis, not accepted uncritically.

A Full Worked Example: Instrumenting shipment-api End to End#

# 1. Ops Agent / OpenTelemetry instrumentation baked into the
# container image (Part 5's Cloud Run deployment)
# 2. Log-based metric for payment failures
gcloud logging metrics create shipment-api-payment-failures \
  --log-filter='jsonPayload.event="payment_failure"'

# 3. Alerting policy on error rate AND the payment-failure metric,
# routed to the shared on-call notification channel
gcloud alpha monitoring policies create \
  --notification-channels=projects/meridian-shipment-prod/notificationChannels/CHANNEL_ID \
  --display-name="shipment-api payment failures"

# 4. Cloud Trace via OpenTelemetry, correlating slow requests to
# specific spans
# 5. Cloud Profiler enabled in the application startup code
# 6. Audit logs (Admin Activity, on by default) already flowing to
# the central security bucket per this chapter's org-wide sink

Real-World Scenario: The Alert Nobody Trusted#

Meridian's original error-rate alert fired so frequently during normal, expected traffic variance that the on-call rotation started reflexively acknowledging and dismissing it without genuinely investigating — an "alert fatigue" pattern that eventually meant a real elevated-error-rate incident went uninvestigated for twenty minutes because it looked identical to the dozens of false alarms the team had already learned to ignore. The immediate cause was a threshold set too sensitively relative to shipment-api's actual normal variance; the deeper cause was that the threshold had never been validated against real traffic data before being deployed, unlike Part 4's autoscaling policies, which were tuned against measured behavior from the start.

The fix mirrored Part 4's own autoscaling-noise lesson: the team pulled two weeks of real error-rate data, computed the genuine normal variance, and set the alert threshold meaningfully above that baseline rather than at a round, arbitrary number picked without reference to real traffic. An alert that fires too often is not a safety margin — it's a slow-motion failure of the alert's actual purpose, since a human who's learned to dismiss an alert without looking stops providing the exact safety net the alert exists to guarantee.

Observability Terminology Map#

Where this mapping holds up well: observability tooling has converged across clouds more than almost any other category this course has compared — Prometheus compatibility specifically is now a shared, portable standard across all three providers, meaning a team's PromQL knowledge transfers directly regardless of which cloud eventually hosts the workload.

How the Five Cloud Operations Products Fit Together#

Five products, each answering a genuinely different question: Monitoring answers "is something wrong right now," Logging answers "what exactly happened," Trace answers "where did the time go in this request," Profiler answers "where does this application spend its own CPU/memory," and Error Reporting answers "what's breaking and how often." A mature observability setup uses all five together, each for the question it actually answers — reaching for Logging alone to answer a Trace-shaped question (as this chapter's practice problems highlight) is a common, avoidable inefficiency.

Pre-Flight Checklist: Is This Observability Setup Production-Ready?#

• Every alert routes to the same trusted on-call channel, not a separate per-product notification surface
• Alert thresholds validated against real historical data, not picked as a round arbitrary number
• Log-based metrics exist for application-level events that native infrastructure metrics can't capture
• Data Access audit logs enabled deliberately for services carrying sensitive data, not reflexively everywhere
• Distributed tracing instrumented across every service in a multi-service request path
• Cloud Profiler enabled given it's free and low-overhead — no reason to skip it
• A saved-query habit exists for any Logs Explorer query used more than once

Common Mistakes and Interview Traps#

Worked Practice Problems#

1. shipment-api's error-rate alert has been firing 3-4 times a week for months, and the on-call team has started dismissing it without investigating each time. A genuine elevated-error-rate incident goes undetected for twenty minutes as a result. What's the actual root cause, and what's the fix?

The root cause is alert fatigue caused by a threshold that was never validated against real traffic data — firing routinely during normal variance trained the on-call team to stop treating the alert as meaningful, which is exactly what let a genuine incident hide among the noise. The fix is pulling real historical error-rate data, computing actual normal variance, and resetting the threshold meaningfully above it — restoring the alert's actual signal value rather than adding a process fix (like requiring investigation of every alert) that doesn't address the underlying miscalibration.

2. A team wants to know exactly which part of a slow multi-service request is responsible for the latency — the database, an external API call, or application logic. Which tool answers this, and why wouldn't Cloud Logging alone be sufficient?

Cloud Trace, via distributed tracing instrumentation — it shows a request's full path across services as a waterfall, with each span's individual duration, directly answering "which part took how long." Cloud Logging alone shows discrete log entries but doesn't inherently connect them into a single request's timing picture across service boundaries — correlating trace IDs with logs (mentioned in Part 4) helps connect the two, but the actual timing breakdown itself comes from Trace's span data, not from reading logs in isolation.

3. Meridian wants to enable Data Access audit logs to strengthen its security posture but is concerned about cost. What's the right scoping approach per this chapter's guidance, and why is "enable everywhere" the wrong default?

Enable Data Access logs deliberately, scoped to the specific services actually handling sensitive data (Cloud Storage buckets with shipment documents, the orders database) rather than blanket-enabling them across every GCP service in the project. Data Access logs, unlike the always-on and free Admin Activity logs, can generate substantial volume and real cost — "enable everywhere" trades a diffuse, expensive logging posture for a security benefit that's actually concentrated in a much smaller set of genuinely sensitive services, meaning the broad approach pays for coverage that adds little real security value beyond the scoped approach.

4. route-optimizer's BigQuery-fed dashboard shows a slow monthly aggregation query, but nobody can tell which specific stage of the query pipeline is slow — the query itself, an upstream Cloud Run function feeding it, or the BigQuery job execution. Which of this chapter's five products actually answers this, and why not just add more logging?

Cloud Trace, provided the whole pipeline is instrumented with distributed tracing spans across each stage (the Cloud Run function, the BigQuery job trigger, and the query execution itself) — it directly shows which span in the chain consumes the most time, the exact multi-service timing breakdown question Trace is built for. Adding more logging would surface more individual events but wouldn't inherently connect them into one coherent timing picture across service boundaries the way a trace waterfall does natively — logging answers "what happened," not "how long did each stage take relative to the others," which is specifically what this symptom needs answered.

5. Meridian's security team asks whether enabling Data Access audit logs for every Cloud Storage bucket, including ones with no sensitive data, would meaningfully improve the company's security posture. What should the platform team's answer be, based on this chapter's guidance?

No — enabling Data Access logs broadly across buckets with no genuinely sensitive content adds real logging volume and cost while providing little additional security value, since the actual risk (unauthorized access to sensitive shipment/customer data) is concentrated in a small, identifiable set of buckets. The right answer is scoping Data Access logging specifically to buckets that actually hold sensitive content (meridian-shipment-assets, say) — the same "enable deliberately, not reflexively" principle this chapter applies to Data Access logs generally, since broad-but-shallow coverage is a worse trade than narrow-but-complete coverage of the resources that actually matter.

Summary and What's Next: Where This Course Goes From Here#

This chapter closed the loop on everything this course built: Cloud Monitoring and alerting tuned against real data rather than guesses, Cloud Logging's router/sink/export architecture, the full observability toolkit (Trace, Profiler, Error Reporting, Managed Prometheus), and the audit-log distinctions that matter for both security and cost. Meridian's platform now has real, working observability across every layer this course covered — compute, GKE and serverless, data, and networking — not just correct configuration that nobody can verify is actually working.

This is the end of GCP Cloud Engineer Foundations. You've built a complete, production-grounded picture of the Associate Cloud Engineer exam's full scope: resource hierarchy and IAM, compute and serverless platforms, storage and databases, networking, and operations — all through one consistent running example rather than disconnected, one-off snippets. From here, this site's five remaining GCP courses go deeper into specific professional tracks: GCP DevOps & CI/CD Platform and GCP SRE & Observability (together covering the Professional Cloud DevOps Engineer exam), GCP Network Engineering (Professional Cloud Network Engineer), GCP Security Engineering (Professional Cloud Security Engineer), and GCP Architecture & Design (Professional Cloud Architect) — each building on the foundation this course just finished, the same way Meridian Logistics' own platform team keeps building on the landing zone Part 1 first bootstrapped.