A decade ago we fought to put every engineer on the pager. You build it, you run it, and you feel the 3am pain your code causes. It worked. Now AI agents triage the alert, find the bad deploy, roll it back and write the summary before anyone wakes up. The pager has gone quiet, and that is a genuine win. It also means the muscle we spent ten years building is quietly wasting away, and most engineering leaders haven't noticed.
I spent years at Amazon where on-call was the sharpest teacher in the building. You learnt a system's real architecture, not the diagram, by debugging it at 2am with a customer-facing dashboard in the red. At Dovetail this year we have cut operational load by around 80%, largely through automation and AI-assisted remediation. I wouldn't trade that back. But I have started asking a question that would have sounded absurd five years ago: how does an engineer get good at incidents when there are hardly any incidents left?
The irony nobody budgeted for
In 1983 Lisanne Bainbridge wrote a short paper called Ironies of Automation. Her argument was simple. Automate the routine work and you leave humans with only the rare, strange, high-stakes failures, while removing the practice that made them capable of handling those failures. Aviation learnt this decades ago. Software is learning it now.
The incidents an AI agent can't resolve are, by definition, the hard ones. The novel failure mode, the cascading dependency, the data corruption that doesn't throw an error. These land on an engineer who last touched a live incident months ago, in a system that has since changed underneath them, with tooling they have mostly watched an agent use. That is not a skills gap you can close mid-incident.
This is a leadership problem, not an engineering one. When the pager goes quiet, the natural management response is to celebrate the metric and quietly reallocate the time. The rotation shrinks, the game days get cancelled because "we haven't had a real incident in ages", and operational readiness becomes an assumption rather than a capability. Quiet is the outcome we wanted. Complacency is the outcome we get by default.
Practice deliberately what you used to practise by accident
Noisy on-call gave us free training. We now have to pay for it on purpose, and the good news is that most of the alternatives are better training than random 3am pages ever were.
Start by treating the agent's work as reviewable output. Every incident an AI resolves should be read by a human the next morning, in the same way we review a pull request. What did it see, what did it try, why did it choose that rollback? Engineers who read agent-resolved incidents build much of the intuition of having handled them, and they catch the cases where the agent fixed the symptom and buried the cause.
Next, bring back game days and make them harder than reality. If production is quiet, break staging on purpose. Kill a dependency, corrupt a queue, expire a certificate, then switch the agent off and let a pair of engineers work it from first principles. The goal isn't to beat the machine. It is to keep a human who can reason when the machine is confidently wrong.
Operational reviews deserve the same attention. Amazon's weekly ops meetings were not about incidents. They were about the leading indicators: latency percentiles drifting, error budgets being spent, alarms that fired and resolved themselves. Spinning a random service's dashboard and asking its owner to explain every graph is still one of the best mechanisms I know for keeping an engineer close to their system.
Finally, put engineers back into the design side of reliability. Operational readiness reviews before launch, failure-mode analysis on new architecture, deleting alerts nobody acts on, writing the correction-of-error document for a near miss that never paged. These disciplines made incidents rare in the first place, and they keep an engineer's mental model current in a way a quiet pager never will.
Quiet is earned, sharp is maintained
None of this is an argument against automation. Fewer 3am pages means happier engineers and more time spent building. That dividend has arrived, but it comes with a maintenance cost, and leaders own it. If your on-call has gone quiet this year, ask when your team last practiced a hard failure without help, and who on the rotation could lead the next one. If the answer is uncomfortable, you have found your next operational investment. The pager going quiet was the easy part. Staying ready for the day it rings is the job now.
Comments