Instrumenting Telemetry

Assumes you're comfortable with the three pillars of observability and OpenTelemetry fundamentals from Observability Part 1, and with the Ops Agent, Cloud Trace, and basic Managed Service for Prometheus setup at ACE depth from GCP Cloud Engineer Foundations Part 8 — this chapter goes one level deeper into the collection layer itself.

Table of Contents#

1. What This Chapter Covers
2. Every Alert in This Course Depends on This Chapter
3. The Ops Agent, Revisited: What It Actually Collects
4. The OpenTelemetry Collector: Agent/Sidecar vs. Gateway
5. Instrumenting Application Metrics With the OpenTelemetry SDK
6. Telemetry From the Platform, Not the Application
7. Choosing What Instruments What — A Decision Framework
8. Optimizing Logs Before They're Ever Stored
9. Google Cloud Managed Service for Prometheus, In Depth
10. Synthetic Monitors: Probing Before a Real User Does
11. Custom and Log-Based Metrics, One Level Deeper
12. Terminology Map: Telemetry Collection Across AWS, Azure, and GCP
13. A Full Worked Example: Instrumenting gps-ingestion End to End
14. Common Mistakes and Interview Traps
15. Worked Practice Problems
16. Summary and What's Next

What This Chapter Covers#

Everything Part 1 and Part 2 built — SLO burn-rate alerts, error-budget gates, incident-mitigation runbooks — depends on a quiet assumption this course hasn't examined yet: that the underlying telemetry is actually flowing, complete, and not accidentally dropped or too expensive to keep. PCDE's observability objective starts exactly here, with "instrumenting and collecting telemetry" (~25% of the exam combined with the managing/analyzing half Part 4 covers) — the collection layer every other layer of this course sits on top of.

GCP Cloud Engineer Foundations Part 8 already introduced the Ops Agent, basic OpenTelemetry instrumentation, and a first Managed Service for Prometheus PodMonitoring resource at ACE depth — enough to get metrics and traces flowing for a single service. This chapter goes one level deeper into each: how the OpenTelemetry Collector itself is deployed and topology-chosen, what "optimizing logs" actually means before cost becomes a problem (not after), Managed Service for Prometheus's real architecture, and two collection surfaces Foundations never touched at all — synthetic monitors and platform-level telemetry like VPC Flow Logs.

Every Alert in This Course Depends on This Chapter#

Meridian's platform team discovered exactly how load-bearing this layer is three weeks after Part 2's incident-response runbook shipped: a real capacity incident on route-optimizer triggered none of the SLO burn-rate alerts Part 1 built, and initial triage assumed the SLO itself was misconfigured. The actual cause was upstream of the SLO entirely — a Cloud Service Mesh sidecar injection had silently failed on one of three node pools during a routine GKE upgrade two weeks earlier, so roughly a third of route-optimizer's real traffic was never emitting the telemetry the SLO's requestBased SLI depended on. The SLO wasn't wrong; it was computing an accurate answer from an incomplete picture, and nobody had a check for "is telemetry coverage itself complete" separate from "is the SLO's computed value healthy."

What to notice: an SLO is only as trustworthy as the telemetry coverage underneath it, and coverage gaps fail silently — nothing errors out, the SLO just quietly computes an accurate answer to a narrower question than the team thinks it's asking. This chapter's practical closing checklist includes a coverage-verification step specifically because of this incident.

The Ops Agent, Revisited: What It Actually Collects#

Foundations Part 8 deployed the Ops Agent as a fleet-wide policy without going into what it actually does under the hood: it's a single agent bundling two independently-configurable sub-agents — a metrics collector (built on the OpenTelemetry Collector) and a logging collector (built on Fluent Bit) — controlled by one YAML config file at /etc/google-cloud-ops-agent/config.yaml.

# Extending the default Ops Agent config on a gps-ingestion Compute
# Engine instance to also tail an application-specific log file the
# default config doesn't know about — the config Devon actually
# needed to add once he'd traced the coverage gap back to a
# consumer daemon logging to disk instead of stdout
logging:
  receivers:
    gps_consumer_app:
      type: files
      include_paths:
        - /var/log/gps-consumer/*.log
  service:
    pipelines:
      gps_consumer_pipeline:
        receivers: [gps_consumer_app]

The OpenTelemetry Collector: Agent/Sidecar vs. Gateway#

For anything beyond a VM the Ops Agent already covers — a containerized workload, a hybrid/multi-cloud fleet, or an application that wants OTLP-native export instead of the Ops Agent's own collection model — the raw OpenTelemetry Collector (Google publishes its own build, tuned for the Google Cloud exporter) is the mechanism, deployed in one of two topologies with genuinely different operational tradeoffs:

# The Google Cloud exporter config a gateway-pattern collector uses to
# translate OTLP data into Cloud Trace, Cloud Monitoring, and Cloud
# Logging's native formats — the same translation layer, one config,
# for every one of route-optimizer's peer services routed through it
exporters:
  googlecloud:
    project: meridian-shipment-prod
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [googlecloud]
    metrics:
      receivers: [otlp]
      exporters: [googlecloud]
    logs:
      receivers: [otlp]
      exporters: [googlecloud]

🔍 From the Trenches: Meridian's own migration from per-pod sidecars to a shared gateway collector for route-optimizer initially used a single gateway replica "to keep it simple," and a routine node drain during a GKE upgrade took that one replica down for ninety seconds — during which every service routed through it silently dropped telemetry, the exact same failure shape as this chapter's opening incident, just from a different root cause. The immediate cause was a single point of failure in the gateway deployment; the underlying condition was treating the collector itself as infrastructure that doesn't need the same availability discipline (multiple replicas, a PodDisruptionBudget) as the application workloads it observes — an easy category error, since "it's just collecting telemetry" undersells how much everything else in this course depends on it staying up.

Instrumenting Application Metrics With the OpenTelemetry SDK#

Foundations Part 8 instrumented shipment-api for tracing via OpenTelemetry. The metrics side of the same SDK is what actually feeds application-level dashboards and custom SLIs with data no infrastructure-level agent can see — a payment-processing queue depth, a business-logic counter, anything specific to what the application itself is doing internally.

# gps-ingestion-consumer emitting a custom histogram metric for its
# own message-processing duration, and a counter for a business-level
# event (a malformed GPS payload) neither the Ops Agent nor Cloud
# Service Mesh telemetry has any way to see
from opentelemetry import metrics

meter = metrics.get_meter("gps-ingestion-consumer")

processing_duration = meter.create_histogram(
    "gps_ingestion.processing_duration",
    unit="ms",
    description="Time to process one GPS ingestion message",
)
malformed_payloads = meter.create_counter(
    "gps_ingestion.malformed_payloads",
    description="Count of GPS payloads that failed schema validation",
)

def process_message(message):
    start = time.monotonic()
    try:
        validate_and_process(message)
    except SchemaValidationError:
        malformed_payloads.add(1, {"source": message.attributes.get("device_type")})
        raise
    finally:
        processing_duration.record((time.monotonic() - start) * 1000)

Both instruments export through whichever collector topology the previous section established (agent/sidecar or gateway) and land in Cloud Monitoring as native custom metrics — queryable, chartable, and alertable exactly like a native infrastructure metric, and eligible as the underlying data source for a future Service Monitoring requestBased or windowsBased SLI the way Part 1's SLO objects already consume run.googleapis.com metrics.

Telemetry From the Platform, Not the Application#

Two collection sources the exam guide names explicitly that neither Foundations nor this course has covered yet, because they're generated by the platform rather than instrumented by an application:

VPC Flow Logs — every IP-level flow through a subnet's VMs, sampled and aggregated, genuinely useful for a "why is traffic between these two services slow or dropped" investigation Cloud Trace can't answer (Trace sees application-level spans, not raw network flows).

# Enable flow logs on gps-ingestion's subnet with a cost-conscious
# aggregation interval and sampling rate — full 5-second/1.0 sampling
# is the default, but 30s/0.5 cuts log volume dramatically for a
# subnet where flow-level detail is a debugging tool, not a
# continuous monitoring requirement
gcloud compute networks subnets update gps-ingestion-subnet \
  --region=us-central1 \
  --enable-flow-logs \
  --logging-aggregation-interval=interval-30-sec \
  --logging-flow-sampling=0.5

Cloud Service Mesh access logs and telemetry — the same sidecar proxy that computed route-optimizer's mesh-native SLO in Part 1 also emits per-request access logs and RED-style metrics (rate, errors, duration) automatically for every mesh-enrolled service, with zero application code changes — genuinely the fastest way to get baseline telemetry on a newly onboarded GKE service before any custom instrumentation exists.

Choosing What Instruments What — A Decision Framework#

This chapter has now covered five genuinely different collection mechanisms for what can feel like the same underlying goal ("get telemetry"). They're complementary, not competing — the real skill PCDE tests is knowing which one answers which question, not picking a single favorite.

Each layer answers a narrower, more specific question than the one below it — and, per the decision table above, is blind to everything the layer below already covers plus its own gaps, which is exactly why a mature service layers several rather than picking one.

Optimizing Logs Before They're Ever Stored#

The cheapest log entry is the one that never gets ingested at all — an exclusion filter drops a log entry before it's written to any bucket, which is a fundamentally different cost lever than exporting and then filtering downstream. This matters enough to PCDE that "optimizing logs" is its own named exam bullet, not an afterthought to collection.

# Exclude noisy, low-value health-check logs from shipment-api's
# default log sink entirely — these hit Cloud Run's load balancer
# thousands of times a day and carry zero diagnostic value once the
# service is confirmed healthy
gcloud logging sinks update _Default \
  --project=meridian-shipment-prod \
  --add-exclusion='name=exclude-health-checks,filter=httpRequest.requestUrl="/healthz" OR httpRequest.requestUrl="/readyz"'

# Sample, rather than fully exclude, routine 2xx load-balancer
# entries — keeping a representative fraction for traffic-pattern
# analysis while cutting the bulk of the ingestion cost
gcloud logging sinks update _Default \
  --project=meridian-shipment-prod \
  --add-exclusion='name=sample-2xx-lb-logs,filter=resource.type="http_load_balancer" AND httpRequest.status>=200 AND httpRequest.status<300 AND sample(insertId, 0.1)'

Google Cloud Managed Service for Prometheus, In Depth#

Foundations Part 8 stood up a single PodMonitoring resource — enough to scrape one service. The PCDE-depth picture is the full architecture underneath that resource, because "collecting metrics… Google Cloud Managed Service for Prometheus" is named explicitly in the exam guide's telemetry objective.

Two collection modes: managed collection runs Google-operated collectors reading your PodMonitoring/ClusterPodMonitoring resources — the default, lowest-operational-burden path. Self-deployed collection runs your own Prometheus server (or the OpenTelemetry Collector's Prometheus receiver) configured to remote-write into Managed Service for Prometheus's storage backend instead of running its own local TSDB — the path for a team with an existing Prometheus investment who wants Google's storage/query layer without giving up their own scrape configuration.

Both paths land data in the same backend (Monarch, Google's internal time-series store), which is what makes recording rules work identically regardless of collection mode — expressed as one of three Kubernetes-style custom resources depending on scope:

# A recording rule pre-computing route-optimizer's p95 request
# latency across the whole mesh — GlobalRules because this needs to
# aggregate across every project in Meridian's metrics scope, not
# just the one cluster it happens to be authored in
apiVersion: monitoring.googleapis.com/v1
kind: GlobalRules
metadata:
  name: route-optimizer-latency-rules
spec:
  groups:
    - name: route-optimizer-slo-inputs
      interval: 30s
      rules:
        - record: route_optimizer:request_latency:p95
          expr: |
            histogram_quantile(0.95,
              sum(rate(route_optimizer_request_duration_seconds_bucket[5m])) by (le)
            )

Synthetic Monitors: Probing Before a Real User Does#

A synthetic monitor is a scheduled, scripted probe against your own application's real endpoints and workflows — running from Google's global probe infrastructure on a schedule, whether or not a real user happens to be hitting that path right now. This is meaningfully different from every telemetry source covered so far: everything else in this chapter is passive (it observes real traffic as it happens); a synthetic monitor is active — it generates its own traffic specifically to verify a path works, catching a failure even during a genuine lull in real usage.

# Meridian's synthetic monitor for the checkout flow that regressed
# silently in this course's Part 1 opening incident — deploys a
# Cloud Run function running a scripted multi-step check, on a
# schedule, independent of whether real customers are checking out
# right now
gcloud monitoring synthetic-monitors create checkout-flow-probe \
  --project=meridian-shipment-prod \
  --display-name="Checkout flow — tracking to payment confirmation" \
  --schedule="*/5 * * * *"

💡 The transferable insight: this is the same active-vs-passive distinction air-traffic control draws between reading live radar returns and running a scheduled equipment self-test — passive radar only shows you what's actually in the air right now, while a self-test verifies the equipment works even during a genuine lull in traffic. A checkout flow that regresses at 3am, when almost no real customer traffic exercises it, is exactly the scenario passive telemetry alone can miss for hours.

Custom and Log-Based Metrics, One Level Deeper#

Foundations Part 8 created a simple counter-style log-based metric (counting payment-failure log entries). The depth PCDE actually expects: a distribution-type log-based metric, which extracts a numeric value from each matching log entry rather than just counting occurrences — turning a log line into a metric with real percentile/histogram behavior, not just a rate.

# Extract the actual latency value logged in each gps-ingestion
# request's structured log entry, as a distribution metric Cloud
# Monitoring can compute p50/p95/p99 against — genuinely different
# from a counter, which can only tell you *how often* something
# happened, not *how large* each occurrence was
gcloud logging metrics create gps-ingestion-request-latency \
  --description="Distribution of gps-ingestion request latencies from structured logs" \
  --log-filter='resource.type="cloud_run_revision" resource.labels.service_name="gps-ingestion-consumer" jsonPayload.latency_ms>0' \
  --value-extractor='EXTRACT(jsonPayload.latency_ms)' \
  --bucket-options=exponential-buckets,num-finite-buckets=64,growth-factor=2,scale=1

Terminology Map: Telemetry Collection Across AWS, Azure, and GCP#

A Full Worked Example: Instrumenting gps-ingestion End to End#

The concrete sequence Devon's team ran after tracing the coverage gap from this chapter's opening incident back to its root cause:

# 1. Ops Agent config extended to tail the consumer daemon's file-based
#    logs (shown earlier), rolled out via VM Manager's OS Config —
#    the same fleet-wide mechanism Foundations Part 4 established
gcloud compute instances ops-agents policies update gps-worker-ops-agent \
  --project=meridian-shipment-prod --zone=us-central1-a

# 2. Verify Cloud Service Mesh sidecar injection is actually present
#    on every node pool — the specific check this chapter's opening
#    incident was missing entirely
kubectl get pods -n route-optimizer -o jsonpath='{.items[*].spec.containers[*].name}' | grep -c istio-proxy

# 3. GlobalRules recording rule for cross-cluster p95 latency (shown
#    earlier), applied via kubectl
kubectl apply -f route-optimizer-latency-rules.yaml

# 4. Synthetic monitor for the checkout flow (shown earlier)
gcloud monitoring synthetic-monitors create checkout-flow-probe \
  --project=meridian-shipment-prod --schedule="*/5 * * * *"

# 5. Distribution log-based metric for gps-ingestion latency
#    (shown earlier)
gcloud logging metrics create gps-ingestion-request-latency \
  --log-filter='resource.type="cloud_run_revision" jsonPayload.latency_ms>0' \
  --value-extractor='EXTRACT(jsonPayload.latency_ms)'

🧪 Hands-on checkpoint: run step 2's kubectl check against a real GKE deployment and confirm the sidecar-proxy count matches the pod count exactly — any mismatch is the same silent coverage gap this chapter's opening incident hit, catchable in seconds instead of the two weeks it took Meridian's team to trace it back.

Common Mistakes and Interview Traps#

Worked Practice Problems#

Problem 1: route-optimizer's Cloud Service Mesh SLO shows healthy 99.9% compliance, but a real capacity incident goes completely undetected by any alert. What's the most likely root cause given this chapter's opening incident, and how would you check for it?

Answer: A silent telemetry coverage gap — some fraction of real traffic isn't emitting the mesh telemetry the SLO depends on (a missing sidecar injection on a subset of nodes, in Meridian's actual case), so the SLO is computing an accurate number from an incomplete sample rather than being wrong outright. The check is direct: confirm the sidecar-proxy container count matches the pod count across every node pool, rather than trusting the SLO's healthy-looking output as proof that telemetry itself is complete.

Problem 2: A team deploys a gateway-pattern OpenTelemetry Collector as a single replica to keep the setup simple, and a routine node drain takes it down for ninety seconds. What's the blast radius, and what's the fix?

Answer: Every workload routed through that single collector silently drops telemetry for the outage's duration — not just the node being drained, since the gateway pattern centralizes collection for potentially many services. The fix is running the collector with the same availability discipline as any other production workload it observes — multiple replicas behind a PodDisruptionBudget — treating "it's just collecting telemetry" as exactly as load-bearing as the services it instruments, not a lesser concern.

Problem 3: An engineer proposes setting up a central Prometheus federation server to aggregate metrics across Meridian's GKE clusters, now that Managed Service for Prometheus is in use everywhere. Should this be built, and why?

Answer: No — federation is explicitly unnecessary with Managed Service for Prometheus, because every self-deployed collector already writes into the same shared global backend (Monarch). A cross-cluster aggregation is already achievable with a single query or a GlobalRules recording rule scoped to the whole metrics scope, without standing up and operating a separate federation layer that would add real complexity to solve a problem this architecture doesn't have.

Summary and What's Next#

This chapter went one layer beneath everything Parts 1 and 2 built: the Ops Agent's actual sub-agent structure, the OpenTelemetry Collector's agent/sidecar and gateway topologies (and the real availability cost of choosing the wrong one), platform-level telemetry sources (VPC Flow Logs, Cloud Service Mesh access logs) alongside the application-level sources Foundations already covered, log optimization before ingestion rather than after, Managed Service for Prometheus's real managed-vs-self-deployed architecture and its Rules/ClusterRules/GlobalRules recording-rule hierarchy, synthetic monitors as the active complement to passive telemetry, and distribution-type log-based metrics. Meridian's coverage gap — an SLO computing a healthy-looking answer from an incomplete picture — is now a checkable condition, not an invisible one.

Part 4 picks up where telemetry collection ends and analysis begins: managing and querying the logs this chapter's collection layer produces at real scale, building dashboards and alerts on the metrics it emits, and the specific GCP mechanisms — Log Analytics, dashboards-as-code, third-party alert routing — that turn raw collected telemetry into something a human can actually act on quickly.