Monitoring & Governance

What it is and where it fits 🎯#

Azure Monitor (metrics, alerts, Log Analytics) is Azure's built-in observability stack; Azure Policy is Azure's org-wide governance engine (enforcing tagging, region restrictions, SKU limits, and more); resource locks and cost management round out the "make sure production stays production" toolkit this page covers. None of this is a substitute for a dedicated APM/logging stack (Datadog, Grafana/Prometheus) if your org already runs one — but Azure Monitor is what's available with zero extra infrastructure, and Azure Policy has no real substitute for Azure-native governance, since it's the only mechanism that can actually deny a non-compliant resource creation before it happens.

Core concepts: how governance actually gets enforced#

A policy definition is just a rule sitting in your subscription doing nothing on its own; it only takes effect once a policy assignment attaches it to a scope (subscription, resource group, or management group). The same definition can be assigned at multiple scopes with different parameters — this is how one "require an environment tag" rule can enforce different allowed values in a prod subscription vs. a sandbox one.

Listing available metrics for a resource#

az monitor metrics list-definitions --resource <resource-id>
az monitor metrics list --resource <resource-id> --metrics "Percentage CPU" --interval 5m
az monitor metrics list --resource <resource-id> --metrics "Percentage CPU" --aggregation Average Maximum --start-time 2026-08-28T00:00:00Z --end-time 2026-08-29T00:00:00Z

az monitor metrics list-definitions tells you which metric names, aggregation types, and dimensions a given resource actually supports before you query it — resource types expose different metrics, so guessing a metric name (or its exact capitalization — Azure metric names are case-sensitive strings like "Percentage CPU", not an enum) is a common source of an empty result with no error.

Creating a metric-based alert#

az monitor action-group create --resource-group my-rg --name my-action-group \
  --short-name myag --action email oncall oncall@example.com

az monitor metrics alert create \
  --resource-group my-rg --name high-cpu-alert \
  --scopes /subscriptions/<sub-id>/resourceGroups/my-rg/providers/Microsoft.Compute/virtualMachines/my-vm \
  --condition "avg Percentage CPU > 80" \
  --window-size 5m --evaluation-frequency 1m \
  --action my-action-group \
  --severity 2

--condition uses a compact expression syntax ({avg,min,max,total,count} METRIC {operator} THRESHOLD) rather than a JSON body — az monitor metrics alert condition create can build more complex dynamic- threshold conditions if a static threshold isn't enough. An action group (created separately, referenced by name/ID in --action) is what actually notifies someone or triggers automation — email, SMS, a webhook, an Azure Function, a Logic App — decoupled from the alert rule so the same action group can be reused across many alerts.

Diagnostic settings: routing logs and metrics off the resource#

az monitor diagnostic-settings create \
  --resource <resource-id> --name send-to-log-analytics \
  --workspace <log-analytics-workspace-resource-id> \
  --logs '[{"category": "AuditEvent", "enabled": true}]' \
  --metrics '[{"category": "AllMetrics", "enabled": true}]'
az monitor diagnostic-settings list --resource <resource-id>

Most Azure resources retain their own activity/audit logs for a limited window and don't emit them anywhere else by default — a diagnostic setting is what routes them somewhere durable and queryable: a Log Analytics workspace (for KQL querying, shown below), a Storage Account (cheap long-term archival), or an Event Hub (streaming to a third-party SIEM). Without one configured, "what happened to this resource last month" is often simply unanswerable after the retention window passes.

Querying a Log Analytics workspace#

az monitor log-analytics workspace create --resource-group my-rg --workspace-name my-workspace
az monitor log-analytics workspace show --resource-group my-rg --workspace-name my-workspace --query customerId -o tsv

az monitor log-analytics query \
  --workspace <workspace-customer-id> \
  --analytics-query "AzureActivity | summarize count() by bin(TimeGenerated, 1h)" \
  --timespan P1D

The --workspace value is the workspace's customer ID (a GUID), not its resource name — get it with the show --query customerId command above. --analytics-query takes a KQL (Kusto Query Language) query, the same language used in the Log Analytics portal blade and in Application Insights; --timespan uses ISO 8601 duration/interval syntax (P1D = 1 day, PT1H = 1 hour).

Sample query output shape (representative — actual column names depend on the query's own summarize clause):

[
  { "TimeGenerated": "2026-08-28T00:00:00Z", "count_": 142 },
  { "TimeGenerated": "2026-08-28T01:00:00Z", "count_": 98 }
]

Real-world scenario: diagnosing an intermittent 503 with KQL#

A production app started returning intermittent 503s with no obvious pattern in the metrics dashboard. The fix started with correlating request failures against deployment events in the same window:

az monitor log-analytics query --workspace <workspace-customer-id> \
  --analytics-query "
    AppRequests
    | where TimeGenerated > ago(2h)
    | where ResultCode == 503
    | summarize FailureCount = count() by bin(TimeGenerated, 5m)
    | order by TimeGenerated desc
  " --timespan PT2H

Cross-referencing the 5-minute failure buckets against az monitor activity-log list --resource-group my-rg --start-time <window-start> surfaced a rolling AKS node pool upgrade running in the same window — the 503s were pods being drained during the upgrade's rolling replacement, not an application bug. Two levels deep: the symptom was 503s; the immediate cause was pods terminating mid-request; the underlying condition was a PodDisruptionBudget that wasn't actually enforcing a minimum-available count during the node pool upgrade, so the drain moved faster than the readiness probes could redirect traffic away in time.

Azure Policy: definitions and assignments#

az policy definition create \
  --name require-tag-environment \
  --rules @policy-rule.json --params @policy-params.json \
  --mode Indexed --display-name "Require an environment tag"
az policy definition list --query "[?policyType=='Custom']" --output table

az policy assignment create \
  --name enforce-env-tag --policy require-tag-environment \
  --scope /subscriptions/<sub-id>/resourceGroups/my-rg

--rules (a JSON policy rule document — an if/then structure evaluating resource properties) and --mode Indexed (evaluate only resource types that support tags/location, vs. All, which also covers resource groups and subscriptions themselves) are the two arguments worth understanding before writing a custom definition; Azure ships hundreds of built-in definitions (az policy definition list --query "[?policyType=='BuiltIn']") covering the common cases (require a tag, restrict allowed locations/SKUs, require HTTPS-only storage) — check those before authoring a custom one from scratch.

Checking policy compliance#

az policy state list --resource-group my-rg --filter "complianceState eq 'NonCompliant'"
az policy state trigger-scan --resource-group my-rg --no-wait
az policy state summarize --resource-group my-rg

Compliance state is evaluated on a periodic cycle (roughly every 24 hours) plus on resource create/update — az policy state trigger-scan forces an on-demand re-evaluation instead of waiting for the next cycle, useful right after creating or changing an assignment when you want to confirm it's working immediately rather than tomorrow.

Resource locks#

az group lock create --resource-group my-rg --name protect-prod --lock-type CanNotDelete
az group lock create --resource-group my-rg --name freeze-prod --lock-type ReadOnly
az group lock list --resource-group my-rg --output table
az group lock delete --resource-group my-rg --name protect-prod

CanNotDelete allows normal read/modify operations but blocks deletion — the common default for a production resource group. ReadOnly is far more disruptive: it blocks any write operation, including ones a running application may perform routinely (e.g. a storage account's own internal operations), so apply it deliberately and expect to remove it before any planned change, not leave it on by habit.

Cost visibility#

az consumption usage list --start-date 2026-08-01 --end-date 2026-08-29 --output table
az consumption budget create --resource-group my-rg --budget-name monthly-cap \
  --amount 5000 --time-grain Monthly --category Cost \
  --start-date 2026-08-01 --end-date 2027-08-01

az consumption is explicitly marked preview/in-development by Azure CLI's own help output — treat its exact output shape as more likely to change than the stable command groups elsewhere on this page, and cross-check anything cost-critical against the Cost Management blade in the Portal before relying on it for a report.

Real-world scenario: enforcing a mandatory tagging standard org-wide#

A finance team needed every resource tagged with a cost-center for chargeback, and manual reminders weren't working:

• Write (or find a built-in) policy definition requiring the CostCenter tag on resource creation
• Assign it with effect deny at the management-group or subscription level, not per resource group one at a time
• Run az policy state trigger-scan immediately after assignment to catch existing non-compliant resources, not just future ones
• Review az policy state list --filter "complianceState eq 'NonCompliant'" weekly until the backlog of pre-existing untagged resources is cleared
• Only then flip a stricter built-in initiative (a bundle of related policies) on top, once the team is used to the single-tag requirement

CI/CD integration recipe: policy-as-code in GitHub Actions#

# .github/workflows/deploy-policy.yml
name: Deploy governance policy
on:
  push:
    branches: [main]
    paths: ['policies/**']

permissions:
  id-token: write
  contents: read

jobs:
  deploy-policy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: azure/login@v2
        with:
          client-id: ${{ secrets.AZURE_CLIENT_ID }}
          tenant-id: ${{ secrets.AZURE_TENANT_ID }}
          subscription-id: ${{ secrets.AZURE_SUBSCRIPTION_ID }}
      - run: |
          az policy definition create --name require-tag-environment \
            --rules policies/require-tag-rule.json --params policies/require-tag-params.json \
            --mode Indexed --display-name "Require an environment tag"
          az policy assignment create --name enforce-env-tag \
            --policy require-tag-environment --scope /subscriptions/${{ secrets.AZURE_SUBSCRIPTION_ID }}

Treating policy definitions as version-controlled JSON files reviewed through a pull request, rather than console clicks, is what makes governance itself auditable — the same "config as code" argument that applies to infrastructure applies doubly to the rules governing infrastructure.

Common pitfalls#

• Guessing a metric name instead of checking list-definitions first — names are case-sensitive strings, and a typo returns an empty result with no error, not a helpful failure.
• Assuming policy compliance is evaluated instantly — the default cycle is roughly 24 hours; use trigger-scan right after an assignment change if you need to confirm it immediately.
• Applying a ReadOnly lock without checking for resources that write to themselves routinely — see the WARNING above.
• Treating az consumption output as production-grade for financial reporting — it's explicitly preview tooling; reconcile against the Cost Management Portal blade for anything that matters.
• Writing a diagnostic setting after an incident instead of before — there's no retroactive backfill.

Exit codes#

0 success · non-zero on any API/validation error — az policy assignment create with a malformed --policy reference or an invalid --scope fails fast with a clear message; a deny-effect policy blocking a later, unrelated resource create is not itself an az policy command failure, it surfaces as a failure on whatever command tried to create that resource.

When to reach for something else#

For anything beyond basic metrics/alerting, most teams still run a dedicated observability stack (Datadog, Grafana + Prometheus/Mimir, or an APM tool) fed by Azure Monitor's diagnostic-settings export rather than building dashboards natively in Azure Monitor — Azure Monitor's real strength is being the source every other tool can pull from via diagnostic settings, not necessarily the dashboard layer itself. For policy authoring at scale, Azure's own Policy as Code guidance and tools like the azure/policy Bicep registry modules are worth adopting once you're maintaining more than a handful of custom definitions by hand.