Track Waiting, Running, Completed, and Failed

Waiting, running, completed, failed. Plus the two that matter just as much across a boundary: stale, and last successful.

Most interfaces show a number and a spinner, which collapses all of them. A person looking at that screen cannot tell whether what they see is current, an hour old, or the surviving fragment of a run that mostly failed.

And the rule that makes the whole thing usable during an incident: a failed run must never erase the last successful result. Yesterday's numbers, clearly labelled as yesterday's, are far more useful than a blank panel at exactly the moment somebody needs numbers.

Job history, not just record history

Record history tells you what changed. It does not tell you which run changed it, who started that run, how long it took, or what else it touched at the same time.

So log the job as well as the record. Every scan, evaluation, and batch, with its progress, its duration, and who ran it. The reason is one sentence: so a bad outcome can be traced back to the run that produced it.

That sentence is what turns an incident from an archaeology project into a lookup. Without it, the question “what else did that job do” has no answer, and you end up checking everything.