A threshold that "seems reasonable" is a guess.
Zero setup for basic metrics — but log-based metrics turn any application-level pattern into an alertable signal.
An error-rate alert fired so often during normal variance that on-call started dismissing it reflexively. A real incident then went uninvestigated for 20 minutes — indistinguishable from the noise.
Fix: pull real historical data, compute genuine normal variance, set the threshold meaningfully above it — the same discipline already applied to autoscaling metrics. Route every alert through ONE trusted paging channel, not a separate surface per product.