I had a fleet of scheduled jobs humming along in the background. The kind of setup you build once and feel clever about for a while. Weeks later I found out three of them were broken.
One had been silenced by a bad import - it loaded, threw nothing visible, and produced nothing. One was erroring several times a day into a path that no longer existed. One was quietly falling back to a default because an identifier had gone stale, doing the wrong work on purpose and calling it done.
No alarm. No trace. They were running. They were just running broken, and nothing told me.
That is the part that should bother you. Not that things broke - things always break. It is that they broke silently, and silence was the exact thing I had trained myself to read as success.
The fantasy in "set it and forget it"
When you wire up an autonomous job, you build a quiet contract with yourself: I did the work, so now I get to stop paying attention. That is the whole appeal. Autonomy is supposed to buy back your attention.
But here is the trap. The signal you use to confirm a job is healthy is usually the absence of a complaint. No error in your inbox. No red on a dashboard you never open. No one tapping you on the shoulder. You read that quiet and you think: still working.
The quiet means nothing. A job that crashed on import is also quiet. A job whose output path vanished is also quiet, right up until someone downstream asks where the report went. The absence of a complaint is not the presence of health. It is just the absence of a complaint.
This is the gap I keep coming back to with the spine - my operator OS, the thing that runs the fleet of background jobs across all my orbits. The spine made it trivial to launch autonomy. It did nothing, on its own, to make that autonomy observable. Those are two different problems, and I had only solved the first one.
The dangerous failures are the ones that look like success
Loud failures are a gift. A job that dies hard, pages you, and stops is doing you a favor - it is telling the truth about its own state. You will fix it within the hour.
The failures that hurt are the ones wearing the costume of success. The fallback that runs clean on a stale identifier. The task that completes, returns zero rows, and exits zero because zero rows is technically a valid result. The import error that nobody catches because the job's only job was to be quiet when things were fine.
Every one of my three was a quiet failure. None of them threw anything I would have seen. Each one looked, from the outside, exactly like a job doing what it was told. The cost was not the break itself - it was the weeks of confident wrongness in between, the decisions I made on top of data I assumed was fresh.
If you only catch the loud failures, you have built a system that protects you from the cheap problems and exposes you to the expensive ones.
Build the thing that turns silence into a signal
The fix is not heroic. It is a tool, and it has a grim and perfect name: a dead-man's-switch.
The idea is simple. The job is required to emit a healthy heartbeat on a cadence - a small "I ran, and I ran correctly" signal at the end of a successful pass. Something separate watches for that heartbeat. If the heartbeat shows up on schedule, you hear nothing. If it goes missing, the watchdog alarms.
Read what that does. It inverts the meaning of silence. Before, silence meant "probably fine." Now, silence from the job triggers noise from the watchdog. The thing you cannot see - a job that died without the courtesy of an error - becomes the thing that is loudest. You have turned an absence into an alarm.
Two details matter. First, the heartbeat has to fire only on real success, not on "the process reached the end." A job that falls back to a default and finishes should not get to send a healthy signal. Tie the heartbeat to the outcome you actually wanted. Second, the watchdog has to live outside the system it watches. A monitor that runs inside the same broken process dies with it, quietly, and you have just built a second silent failure.
And then the part nobody wants to hear: you have to check the alert on a schedule too. "Set it, wire it to an alert, and check the alert" is the honest version of the phrase. The watchdog is not a magic spell. It is a tripwire, and a tripwire only works if someone walks the perimeter.
The reframe
I used to think the work of autonomy was building the jobs. It is not. Building the job is the easy half. The real work is building the system that tells you, without you asking, when a job has stopped telling the truth about itself.
An autonomous system you can observe is an asset. It does work while you sleep and it tells you when it can't. An autonomous system you cannot observe is something else entirely. It is a liability you have stopped looking at - generating confident output, drifting from correct, and wearing the same calm face whether it is right or catastrophically wrong.
The clever part was never the fleet. The clever part is the watchdog that makes the fleet honest. Build that first, or admit you are not running autonomy. You are just running unattended.
Field Notes from the Agentic Operator is a personal series. These are my own views, not those of my employer or any organization I work with, and nothing here relies on non-public information.
← All writing