Monitoring automation you can't watch manually
There's a specific irony in a lot of first-generation automation: it's built to remove a manual task, and then someone ends up manually checking every morning that the automation actually ran. That's not automation, it's just moved the manual work one layer up. Good monitoring is what actually lets a team stop watching, and it's usually the most under-invested part of the build.
Three layers, not one
Most teams stop at the first layer: did the job run without throwing an error. That catches crashes but misses almost everything else that can go wrong. We build monitoring in three layers, and the second and third catch the failures that actually cause damage.
- Did it run? The basic layer: a crash, a timeout, a job that never fired. Easy to catch, rarely the expensive failure.
- Did it do the right amount of work? Row counts, record counts, file sizes compared against a normal range. Catches partial failures the first layer misses entirely.
- Did the output make sense? Sanity bounds on the actual values, a price that shouldn't be negative, a quantity that shouldn't jump 10x overnight. Catches the failures that would otherwise reach a customer.
Alert fatigue kills monitoring faster than bugs do
The failure mode on the other side is just as common: monitoring that alerts on everything, so the person receiving it mutes the channel within a month and the whole system becomes theater. Every alert should map to an action someone can actually take. If an alert exists that nobody has ever acted on, it's noise, and it's quietly training people to ignore the ones that matter.
Monitoring isn't there to tell you everything is fine. It's there so you don't have to ask.
The test that actually matters
Before calling monitoring done, we ask one question: if this pipeline silently started producing wrong output tomorrow, how long before someone finds out, and how do they find out. If the honest answer is 'whenever someone happens to notice the numbers look off,' the automation isn't finished, regardless of how well the happy path works.