Monitoring, Logging & Operations

Assumes you're comfortable with every prior chapter's compute, storage, and networking resources, this final chapter is how you actually know whether all of it is healthy.

Table of Contents#

1. What This Chapter Covers
2. Cloud Monitoring: Alerts and Custom Metrics
3. Cloud Logging: Buckets, Log Analytics, and Routers
4. Audit Logs, Firewall Logs, and VPC Flow Logs
5. Viewing and Filtering Logs
6. Ops Agent and Managed Service for Prometheus
7. Cloud Diagnostic Tools: Trace, Profiler, and Query Insights
8. Personalized Service Health
9. Gemini Cloud Assist for Monitoring and Active Assist
10. Cloud Hub: One View Across Events, Health, and Compliance
11. A Full Worked Example: Meridian's Observability Stack
12. Real-World Scenario: The Alert Nobody Trusted Anymore
13. Second Real-World Scenario: The Slow Query Query Insights Had Been Flagging for Weeks
14. Part 8 gcloud Cheat Sheet
15. Pre-Flight Checklist: Is This Observability Setup Production-Ready?
16. Common Mistakes and Interview Traps
17. Worked Practice Problems
18. Summary: The Whole Series, Connected

What This Chapter Covers#

🎯 By the end of this chapter, you'll be able to build an observability stack that answers three questions fast: is something wrong right now (Monitoring), what actually happened (Logging), and why (the diagnostic tools), plus know which of GCP's several 2026-era AI-assisted operations surfaces to reach for.

Cloud Monitoring: Alerts and Custom Metrics#

Cloud Monitoring collects metrics from every GCP resource automatically and lets you define alerting policies: a condition (a metric crossing a threshold, an absence of expected data) paired with a notification channel (email, Slack, PagerDuty, Pub/Sub, a webhook).

gcloud alpha monitoring policies create \
  --notification-channels=CHANNEL_ID \
  --display-name="Route optimizer CPU high" \
  --condition-display-name="CPU above 85% for 5 minutes" \
  --condition-filter='resource.type="gce_instance" AND metric.type="compute.googleapis.com/instance/cpu/utilization"' \
  --condition-threshold-value=0.85 \
  --condition-threshold-duration=300s

Custom metrics let you ingest application-specific or log-derived measurements (orders processed per minute, a queue's actual backlog depth) that GCP's own automatic metrics can't see, since Cloud Monitoring only has visibility into what an application or a log entry explicitly reports.

Cloud Logging: Buckets, Log Analytics, and Routers#

Every log entry lands in a log bucket (the storage container, with its own retention policy), and a log router's sinks decide which log entries go where, a default _Default bucket for general retention, a _Required bucket (admin activity and system event logs, retained 400 days, not user-configurable), or a custom sink routing specific entries to BigQuery, Pub/Sub, or an external system entirely.

Log Analytics lets you run SQL queries directly against a log bucket's contents, treating logs as a queryable dataset rather than something you can only scroll through in the console's log viewer, useful for the kind of aggregate question ("how many 5xx responses did this service return per hour, over the last week") that scrolling through raw entries can't efficiently answer.

Caption: the _Required bucket's 400-day retention isn't configurable even by an organization administrator, a deliberate design choice guaranteeing a minimum audit trail survives regardless of what any project-level retention policy elsewhere is set to.

Audit Logs, Firewall Logs, and VPC Flow Logs#

Three distinct log types, frequently conflated:

Data Access audit logs generate substantially more volume than Admin Activity logs (every single read counts, not just configuration changes), which is exactly why they're not universally enabled by default, and why enabling them broadly is a deliberate cost and retention decision, not a free security upgrade.

Viewing and Filtering Logs#

The Logs Explorer's query language filters by resource type, severity, and any structured field in the log payload:

🔍 A diagnostic habit worth building: when investigating an incident, filter by severity>=ERROR first to find the failure's timestamp, then widen the time window and remove the severity filter to see the full context (what request preceded the error, what happened immediately after), rather than scrolling through unfiltered logs from the start.

Ops Agent and Managed Service for Prometheus#

The Ops Agent is Google's unified agent for VM-based workloads, replacing the older separate Monitoring and Logging agents with a single installed process collecting both metrics and logs. Managed Service for Prometheus runs Prometheus-compatible metric collection without you operating Prometheus's own storage backend, letting a team keep existing PromQL queries and Prometheus-format instrumentation while GCP handles the actual storage and scaling of the time-series data.

Cloud Diagnostic Tools: Trace, Profiler, and Query Insights#

💡 These four tools form a natural investigative sequence for a "why is this slow" question spanning Part 4 through Part 6's stack: Trace narrows a slow request down to which service, Profiler narrows a slow service down to which function, and Query Insights plus the index advisor narrow a slow database call down to which query and which missing index.

Caption: each tool's output decides which tool comes next, Trace's finding of "the database call is the slow part" is what makes Query Insights the right next stop, not a parallel, independent check.

Personalized Service Health#

Personalized Service Health filters Google's own incident and maintenance communications down to what's actually relevant to your projects, instead of the generic public status dashboard listing every incident across every GCP customer worldwide. It supports the same alerting channels as Cloud Monitoring (email, SMS, PagerDuty, Slack, Pub/Sub, webhook) and a dedicated API for programmatic access, letting a team's incident-response tooling ingest a Google-side outage as just another alert in its existing pipeline, rather than a human having to separately remember to check the public status page during an investigation.

Gemini Cloud Assist for Monitoring and Active Assist#

Gemini Cloud Assist, introduced for resource inventory in Part 1, extends into monitoring specifically: as of the 2026 platform update, it supports multi-turn, proactive troubleshooting sessions that can invoke gcloud, kubectl, and Terraform directly, working through an incident's likely causes conversationally rather than requiring you to manually correlate Trace, Profiler, and log data yourself.

Active Assist is a separate, complementary portfolio of proactive recommendations, organized by value pillar (Cost, Security, Performance, Reliability, Manageability, Sustainability), surfaced automatically based on your actual resource usage patterns: an idle VM worth rightsizing, an overly permissive IAM binding, a commitment worth purchasing given sustained usage. Where Gemini Cloud Assist answers a question you actively ask, Active Assist surfaces things worth knowing that you might never have thought to ask about.

Cloud Hub: One View Across Events, Health, and Compliance#

Cloud Hub aggregates operations data that would otherwise require checking several separate consoles: active events (Google-side incidents and maintenance, the same data Personalized Service Health surfaces), Cloud Monitoring health and performance data, Security Command Center's security and compliance posture, and open support cases, all in one dashboard. It's the closest thing GCP offers to a single-pane-of-glass operational view, and its Cloud Asset Inventory integration (Part 1) means the same underlying resource data powering Gemini Cloud Assist's chat answers also feeds Cloud Hub's aggregated health view.

A Full Worked Example: Meridian's Observability Stack#

# 1. Ops Agent on every Compute Engine VM via a startup script or VM Manager OS Policy
gcloud compute instances add-metadata meridian-route-optimizer-1 \
  --metadata=enable-osconfig=TRUE

# 2. An alerting policy tied to Personalized Service Health's channel too
gcloud alpha monitoring policies create \
  --notification-channels=CHANNEL_ID \
  --display-name="Dispatch API error rate" \
  --condition-filter='resource.type="cloud_run_revision" AND metric.type="run.googleapis.com/request_count" AND metric.labels.response_code_class="5xx"'

# 3. A custom sink routing production logs to BigQuery for Log Analytics
gcloud logging sinks create meridian-prod-to-bq \
  bigquery.googleapis.com/projects/meridian-freight-prod-8f2k/datasets/logs_analytics \
  --log-filter='resource.labels.project_id="meridian-freight-prod-8f2k"'

# 4. Query Insights enabled on the Cloud SQL instance from Part 6
gcloud sql instances patch meridian-dispatch-db --insights-config-query-insights-enabled

Real-World Scenario: The Alert Nobody Trusted Anymore#

Meridian's dispatch service had an alerting policy with a threshold set so aggressively low that it fired several times a day, most of them for transient blips that self-resolved within a minute with no real user impact. Over several months, the on-call rotation developed a reflexive habit of acknowledging and dismissing the alert without investigating, since "it's probably nothing" had been true dozens of times in a row. When a genuine, sustained outage eventually did fire the same alert, it was dismissed the same reflexive way, and the real incident ran forty extra minutes before someone happened to notice the dashboard independently. The immediate cause was alert fatigue; the underlying condition was a threshold set without ever being tuned against real historical data, treating "an alert exists" as equivalent to "the alert is actually useful." The fix: every alerting policy at Meridian now goes through a documented tuning pass against at least two weeks of historical metric data before going live, and any policy firing more than a handful of times a month without a corresponding real incident gets re-tuned or retired.

Second Real-World Scenario: The Slow Query Query Insights Had Been Flagging for Weeks#

Meridian's dispatch database gradually slowed over several weeks, with individual page loads growing from a few hundred milliseconds to several seconds, a change gradual enough that no single day's degradation ever crossed an alerting threshold. Query Insights had been correctly flagging the exact culprit query, a join missing an index on a newly added foreign key column, since almost the beginning of the slowdown, visible in its dashboard the entire time. Nobody had looked, because nobody had a habit of checking Query Insights proactively; the team only discovered the flagged query after a customer complaint finally prompted a manual investigation. The underlying condition wasn't a missing tool, the diagnostic data existed and was correct the whole time, it was a missing process: no one had a standing cadence for reviewing Query Insights' top offenders, so a slow, gradual regression that never tripped a hard alert threshold went unnoticed for weeks. The fix was adding the index advisor's specific recommendation, and a new weekly fifteen-minute review of Query Insights' top queries as a standing team ritual, independent of whether any alert had fired.

Part 8 gcloud Cheat Sheet#

Pre-Flight Checklist: Is This Observability Setup Production-Ready?#

• Every alerting policy has been tuned against real historical data, not left at a default or arbitrary threshold
• Data Access audit logs are enabled specifically where compliance or investigation needs justify their volume, not blanket-enabled or blanket-disabled without a decision
• Personalized Service Health feeds the same incident channel as internal Cloud Monitoring alerts
• Query Insights (or the equivalent for the database in use) has a standing review cadence, not just a dashboard nobody checks
• Log retention (_Default, _Required, and any custom sink) matches actual compliance and investigation requirements
• Cloud Trace and Profiler are actually instrumented on production services, not just available in theory

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A team wants to know whether a specific customer's data was read from a Cloud Storage bucket last Tuesday, for a compliance investigation. Which log type answers this, and why might the answer simply not be available?

Answer: Data Access audit logs specifically capture object reads. The answer may not be available if Data Access logging wasn't enabled for Cloud Storage at the time, since it's off by default for most services (due to the sheer log volume every single read would generate) and has to be explicitly turned on ahead of time; it can't retroactively reconstruct access that happened before it was enabled.

Problem 2: A service's response latency has crept up gradually over three weeks, never triggering any single-day alert threshold, until a customer complaint prompts investigation. The root cause turns out to be a missing database index that Query Insights had been flagging the entire time. What process change, not tooling change, would have caught this earlier?

Answer: A standing, scheduled review of Query Insights' top-flagged queries (a short weekly check is a reasonable cadence), independent of whether any alerting threshold has fired. The tooling already had the correct answer visible the whole time; the gap was a missing habit of proactively checking a diagnostic dashboard rather than only reacting to alerts, which by design only fire on a threshold crossing, not on a slow, cumulative drift that never crosses one in a single day.

Problem 3: An on-call engineer investigating a service degradation spends two hours checking their own application's logs, metrics, and recent deployments, finding nothing wrong, before discovering the actual cause was a Google-side incident affecting the underlying managed service. What should have been checked first, and how could this be automated going forward?

Answer: Personalized Service Health should be checked at the very start of any investigation, not after exhausting internal leads, since it specifically surfaces Google-side incidents relevant to the affected project. Going forward, this can be automated by wiring Personalized Service Health's alerting into the same channel used for internal Cloud Monitoring alerts, so a Google-side incident shows up as just another entry in the team's existing incident feed rather than requiring someone to separately remember to check a different dashboard.

Summary: The Whole Series, Connected#

This chapter closed the loop the whole series has been building toward: Cloud Monitoring and Logging give you the raw signal, the diagnostic tools (Trace, Profiler, Query Insights) narrow a problem down to its actual cause, and the 2026-era AI-assisted layer, Gemini Cloud Assist, Active Assist, Personalized Service Health, and Cloud Hub, increasingly does a meaningful share of that correlation work for you, provided the underlying instrumentation and alerting discipline from this chapter is actually in place.

Across all eight parts, Meridian Logistics went from an empty resource hierarchy to a fully operational platform: a structured organization with dry-run-validated policies (Part 1), billing guardrails and the right infrastructure tooling for the job (Part 2), an IAM model built on groups, short-lived credentials, and federation instead of keys (Part 3), a compute fleet on the right disks and machine types (Part 4), serverless and agent workloads scaled correctly (Part 5), a data layer with tested backups and controlled encryption (Part 6), a deliberately designed network using identity-based firewall policy (Part 7), and now, the observability to know when any of it breaks and why. That's the actual job of an Associate Cloud Engineer, and it's also exactly what the exam tests. From here, the next step on this site's certification path is the GCP DevOps & CI/CD Platform course, building the deployment pipelines that ship changes onto the foundation this series just built.