Performance Monitoring & FinOps on GCP

Assumes Part 5's tracing and troubleshooting toolkit — this chapter uses the same observability stack for a different purpose: not finding what's broken, but finding what's expensive and what's slow when nothing is actually failing.

Table of Contents#

1. What This Chapter Covers
2. The Bill That Didn't Match Expectations
3. Cloud Profiler, In Depth
4. Application Performance Monitoring: Tying Trace, Profiler, and Query Insights Together
5. Active Assist Insights for Performance
6. The FinOps Framework: Inform, Optimize, Operate
7. Observability Costs Are Real Infrastructure Costs
8. Spot VMs as a Deliberate Cost Strategy, Not Just an Incident Risk
9. Infrastructure Cost Planning: CUDs, SUDs, and Network Tiers
10. Optimizing Individual Workload Costs: GKE, Cloud Run, Compute Engine
11. Terminology Map: FinOps Tooling Across AWS, Azure, and GCP
12. A Full Worked Example: Meridian's Quarterly FinOps Review
13. Common Mistakes and Interview Traps
14. Worked Practice Problems
15. Summary: Completing GCP SRE & Observability

What This Chapter Covers#

This course's final chapter covers PCDE's last exam section — performance and cost optimization (~12%) — the smallest section by weight but genuinely distinct in character from everything before it: Parts 1 through 5 built the tools to detect and respond to problems, and this chapter uses those same tools (Trace, Profiler, Monitoring) for a proactive question instead — is this system running efficiently, and is Meridian spending the right amount of money to keep it that way. FinOps closes the loop this entire course opened in Part 1, where the opportunity-cost-of-nines discussion first put a real dollar figure next to a reliability decision.

The Bill That Didn't Match Expectations#

Five months into this course's timeline, Meridian's first-ever FinOps review surfaced a real surprise: gps-ingestion's Managed Service for Prometheus cost had grown 4x over two months with no corresponding growth in actual traffic or service count. The cause traced back to a well-intentioned debugging change from Part 5's cross-service incident — an engineer had added a recording rule grouping request-latency by device_id (the specific GPS tracker hardware ID) "to spot a misbehaving device faster next time," and Meridian's fleet had over 40,000 distinct device IDs, each now generating its own permanently-stored time series.

What to notice: nobody made an obviously wrong decision — grouping by device ID for debugging is a reasonable instinct, and it worked exactly as intended for the incident it was built for. The failure was never catching that a debugging-motivated change had a real, compounding cost implication that nobody connected back to the monitoring bill until the quarterly review surfaced it as an unexplained trend. This chapter's own section on observability cost (later) is the direct fix for the specific mechanism behind this incident; the broader lesson — that a FinOps review is what actually catches a cost regression like this one, not vigilance alone — is why this chapter treats FinOps as a recurring discipline rather than a one-time setup task.

Cloud Profiler, In Depth#

Foundations Part 8 enabled Cloud Profiler with two lines of code and moved on. The depth PCDE actually expects: reading what Profiler actually produces — a flame graph, where each bar's width represents either CPU time or, for heap profiles, bytes allocated, and the stack depth shows the call chain that led there.

import googlecloudprofiler
googlecloudprofiler.start(
    service="route-optimizer",
    service_version="v13",
    verbose=3,
)

💡 The transferable insight: a CPU flame graph and Part 5's trace waterfall answer genuinely different questions that are easy to conflate because both are "performance tools." A trace shows where time goes across a request's path through multiple services — the database, an external call, application logic, as black boxes. A CPU flame graph shows where time goes inside the application's own code, function by function — it can tell you that calculate_route_eta() specifically, not just "the application logic step," is consuming 40% of CPU time, a level of detail no trace span alone provides.

Application Performance Monitoring: Tying Trace, Profiler, and Query Insights Together#

PCDE's "application performance monitoring" bullet isn't a fourth new product — it's the deliberate combination of tools this course has already built, applied together. route-optimizer's CPU flame graph from the previous section and Cloud SQL's Query Insights (Foundations Part 8) answer complementary halves of the same investigation: Query Insights shows which database query is slow, and a CPU flame graph on the application side shows whether the application's own code (serialization, business logic before or after the query) is contributing meaningfully to total latency alongside it.

What to notice: this is Part 5's troubleshooting decision tree extended one layer deeper specifically for the "performance/latency issue" leaf — trace narrows to which call, and then either Query Insights or Profiler narrows further to which specific query or function, depending on where that call actually landed.

Active Assist Insights for Performance#

Part 1 introduced Active Assist's cost-category recommendations for service retirement. The performance category is a distinct set of recommendations — predictive and automated suggestions like right-sizing a persistent disk based on genuinely observed I/O patterns, or flagging a Compute Engine instance whose CPU is consistently pegged near 100% (the opposite signal from Part 1's idle-resource cost recommendation, and just as actionable).

# List performance-category recommendations for Meridian's shipment
# infrastructure — the same Recommender API surface Part 1 already
# used for cost, now filtered to a different value category entirely
gcloud recommender recommendations list \
  --project=meridian-shipment-prod \
  --recommender=google.compute.instance.MachineTypeRecommender \
  --location=us-central1-a

The FinOps Framework: Inform, Optimize, Operate#

Everything specific this chapter covers from here forward — billing visibility, CUD strategy, per-workload optimization — fits inside a standard three-phase FinOps operating model, worth naming explicitly because PCDE tests "implementing FinOps practices" as a discipline, not just a pile of individual cost levers.

Inform — Foundations Part 2 already built billing export to BigQuery and a budget alert; the FinOps-maturity version adds a Looker Studio dashboard breaking spend down by label (team, service, environment), so a stakeholder can answer "what does shipment-api cost us this month" without a platform engineer running a manual query. Optimize is where CUD purchases, Recommender-driven rightsizing, and Spot strategy live — the active waste-reduction phase this chapter spends the most time on. Operate is the ongoing discipline: budgets as guardrails (not just alerts after the fact), labeling enforced at resource-creation time (an org policy requiring a team label, say) rather than retrofitted, and a real recurring review cadence — Meridian's team settled on monthly, walking the previous month's Recommender output and billing trend together as one standing meeting.

Observability Costs Are Real Infrastructure Costs#

This course's own instrumentation, built across Parts 3 through 5, is itself a real, growing line item — worth naming explicitly because it's easy to treat observability spend as separate from "real" infrastructure cost, when it's really the same budget.

Spot VMs as a Deliberate Cost Strategy, Not Just an Incident Risk#

Part 2 warned against Spot VMs specifically for emergency incident overflow capacity — a narrow, correct warning that shouldn't be over-generalized into "avoid Spot entirely." For routine, fault-tolerant elastic scaling — exactly the opposite context from an active incident — Spot VMs at up to a claimed ~60-91% discount off on-demand pricing (the exact figure depends on machine type and region) are one of the single highest-leverage cost levers available, provided the workload genuinely tolerates a preemption.

Infrastructure Cost Planning: CUDs, SUDs, and Network Tiers#

Foundations Part 2 already introduced Committed Use Discounts and Sustained Use Discounts as a preview; this chapter's depth is choosing correctly between them and a third lever the exam guide names explicitly that Foundations never covered: network tiers.

# Meridian's own network-tier decision: shipment-api's customer-facing
# tracking API stays on Premium (the default), while the nightly BigQuery
# export job explicitly uses Standard tier — a deliberate split, not a
# single blanket choice for the whole project
gsutil -o "GSUtil:default_project_id=meridian-shipment-prod" \
  mb -c standard -l us-central1 gs://meridian-nightly-exports

🔍 From the Trenches: a different team, migrating a workload off Compute Engine onto GKE over a two-quarter window, purchased a 3-year CUD sized against their pre-migration Compute Engine footprint specifically to "lock in the discount before prices rise." By month four of the migration, over half the committed capacity sat unused as workloads moved to GKE nodes the CUD didn't cover, and the remaining 32 months of the commitment kept billing at the full committed rate regardless. The immediate cause was sizing the commitment against a footprint already known to be shrinking; the underlying condition was treating "lock in savings now" as more urgent than confirming the baseline it was locked against would actually persist — a 1-year CUD, reviewed and re-committed at renewal, would have captured most of the same discount with a fraction of the stranded-commitment risk once the migration's real trajectory became clear.

Optimizing Individual Workload Costs: GKE, Cloud Run, Compute Engine#

The exam guide's closing bullet — optimizing GKE, Cloud Run, and Compute Engine costs specifically — has a genuinely different right answer per platform, worth a dedicated decision framework rather than generic advice.

Cloud Run: request-based vs. instance-based billing. Request-based billing (the default) charges only during actual request processing, startup, and shutdown — no charge while idle. Instance-based billing charges for an instance's entire lifecycle, at a lower per-second rate but for more seconds.

shipment-api's steady, high-volume traffic sits in the "instance-based often cheaper" quadrant — the actual reason Meridian's own production config for it uses instance-based billing rather than the request-based default.

GKE: Autopilot's accurate-requests reward vs. Standard's bin-packing reward. Autopilot bills per pod resource request directly, with no node management — it rewards accurate pod requests (asking for what you actually use), and is genuinely cheaper than an under-utilized Standard cluster. Standard mode bills per node regardless of how tightly pods are packed onto it, rewarding deliberate, active bin-packing — for a dense fleet of many small microservices, a well-bin-packed Standard cluster (especially layered with Spot node pools for fault-tolerant workloads) can undercut Autopilot by an order of magnitude, because Autopilot's per-pod billing model doesn't let many small pods share a node's otherwise-wasted capacity the way tight bin-packing does.

Compute Engine: custom machine types over the nearest predefined shape. A workload that needs 6 vCPUs and 16GB RAM but only fits predefined shapes at 8 vCPUs/32GB pays for capacity it never uses — a custom machine type sized to the real requirement (confirmed via the same MachineTypeRecommender this chapter already queried for performance signal) closes that gap directly.

Terminology Map: FinOps Tooling Across AWS, Azure, and GCP#

A Full Worked Example: Meridian's Quarterly FinOps Review#

The concrete agenda Priya's team now runs every quarter, closing the loop on this entire course:

# 1. Inform: pull the quarter's billing trend by label
bq query --use_legacy_sql=false \
  'SELECT labels.value AS team, SUM(cost) AS total_cost
   FROM `meridian-shipment-prod.billing_export.gcp_billing_export_v1`,
   UNNEST(labels) AS labels
   WHERE labels.key = "team"
   GROUP BY team ORDER BY total_cost DESC'

# 2. Optimize: review every recommender category, not just cost
for recommender in google.compute.instance.IdleResourceRecommender \
                    google.compute.instance.MachineTypeRecommender \
                    google.compute.commitment.CommitmentRecommender; do
  gcloud recommender recommendations list \
    --project=meridian-shipment-prod --recommender=$recommender \
    --location=us-central1-a
done

# 3. Operate: confirm every new resource created this quarter carries
#    the required cost-attribution label, per org policy
gcloud asset search-all-resources \
  --scope=projects/meridian-shipment-prod \
  --query='-labels:team'

🧪 Hands-on checkpoint: run step 3's asset search against a real project and confirm it returns zero resources — any hit is an unlabeled resource that will show up as unattributed spend in the next quarter's Inform step, the exact gap a real FinOps "Operate" discipline exists to close before it becomes a recurring blind spot.

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: A team's Cloud SQL Query Insights shows a specific query is slow, but the application's own CPU flame graph shows a completely different function — a JSON serialization step — consuming 35% of CPU time on the same request path. Which finding should get optimization priority, and why isn't this a contradiction?

Answer: This isn't a contradiction — Query Insights and Cloud Profiler answer genuinely different, complementary questions (which query is slow vs. which application function is CPU-expensive), and both can be real, independent contributors to the same request's total latency. Priority should go to whichever one actually sits on the request's critical path and contributes more to total latency, determined by checking both against the trace waterfall's timing breakdown — not by assuming one tool's finding invalidates the other's.

Problem 2: A team builds a Managed Service for Prometheus recording rule that groups a request-latency histogram by raw request_id, intending fine-grained debugging visibility, and their monitoring bill triples within a month. What happened, and what's the fix?

Answer: Grouping by a high-cardinality label like a raw request ID creates one unique stored time series per distinct request ID — effectively unbounded cardinality growth, since every request generates a new label value. The fix is grouping by a genuinely bounded dimension (region, service name, status code) instead, and reaching for trace-level detail (Part 5) rather than per-request metric cardinality when request-level granularity is actually the goal.

Problem 3: Meridian is deciding between GKE Autopilot and GKE Standard for a new fleet of 40 small internal microservices, each with modest, fairly uniform resource needs. Which should they choose, and why?

Answer: GKE Standard, layered with Spot node pools for the fault-tolerant ones — per this chapter's cost model, Autopilot's per-pod billing doesn't let many small pods share a node's otherwise-wasted capacity the way active bin-packing on Standard does, and a dense fleet of many small, uniform-shaped services is exactly the scenario where deliberate bin-packing wins by a wide margin. Autopilot would be the better choice only if Meridian's team specifically didn't want to take on the operational cost of managing node pools and bin-packing strategy themselves.

Summary: Completing GCP SRE & Observability#

This chapter closed the loop this course opened in Part 1: Cloud Profiler's flame graphs and Query Insights as complementary application-performance tools alongside Part 5's tracing, Active Assist's full five-category recommendation surface (not just cost), a real FinOps operating model (Inform, Optimize, Operate) rather than a pile of disconnected cost tips, observability's own real cost footprint and the cardinality trap that can silently inflate it, Spot VMs correctly scoped to routine elastic workloads rather than incident overflow, CUD/SUD/network-tier tradeoffs grounded in real usage data, and per-platform cost optimization for GKE, Cloud Run, and Compute Engine specifically. Meridian's quarterly FinOps review — pulling real billing data, walking every recommender category, and verifying labeling discipline — is the concrete, ongoing practice this whole course's observability investment ultimately pays for itself through.

This completes GCP SRE & Observability, covering PCDE exam sections 3-5 (applying SRE practices, observability and troubleshooting, and performance/cost optimization) in full. Combined with GCP DevOps & CI/CD Platform, these two courses cover the complete Professional Cloud DevOps Engineer exam — from bootstrapping an organization and building CI/CD pipelines through operating and optimizing what those pipelines ship. Meridian Logistics' platform team, across both courses, has gone from hand-run gcloud commands and a Friday-afternoon incident to a fully observable, error-budget-gated, cost-aware production platform. The next courses in this site's GCP certification track — GCP Network Engineering, GCP Security Engineering, and GCP Architecture & Design — build on this same foundation for their respective Professional certifications.