17 Principle Seven — Red Lines and the Human in the Loop

Automation in regulated domains has one non-negotiable design fact: some decisions belong to humans, and the system must know which ones. The failure mode of over-automation is not dramatic; it is a slow migration of judgment into defaults, until one day nobody can say who decided the thing that everyone assumed someone decided.

The discipline has two halves: red lines that no automation may cross, and loops that return to humans at the right moments.

17.1 Red lines are few, absolute, and written

A red line is a rule that admits no exception under deadline pressure — because the moment a red line has an exception procedure, it is a guideline wearing a costume. In clinical pipelines they tend to look like this:

Patient and study data never cross boundaries they should not cross. Data acquired for one purpose, under one governance regime, never leaks into another — not into a demo environment, not into a training set, not into a vendor’s cloud. Systems enforce this with structure (separate environments, synthetic demo data) rather than policy, because structure survives Friday evenings.
Blind states stay blind. Where a trial’s integrity depends on who knows what, the system enforces the blind in its access paths, and any unblinding is an event recorded with a name attached — never a configuration flag flipped quietly.
The line cannot be talked past. A failed verification gate, a missing signature, a draft without approval: these stop the render. Escalation exists, but it is an explicit, signed, recorded act by a human with authority — not an override parameter.

The test of a red line is organizational, not technical: when a senior person, under real pressure, asks for an exception, what happens? In a healthy system the answer is “here is the deviation form and here is what it will say about who approved what.” In an unhealthy one, someone with root access “fixes it” and the graph learns to look the other way.

17.2 The loop: route judgment to humans

Between red lines, automation should be aggressive — but it must recognize the edges of its own competence. The pattern that works in practice is not “human approves everything” (which produces rubber-stamping, the illusion of oversight) but structured routing: the system classifies each unit of work as within its deterministic competence or not, and the exceptions become a queue of specific, well-framed questions.

A spec the rules engine cannot fully express does not become a silent default; it becomes a review item that says exactly which derivation is under-specified. A number that fails comparison does not become a warning; it becomes a mismatch record with both lineages attached. An AI-generated draft does not become output; it remains labeled draft until a named human accepts it.

Done well, the loop is ergonomic: humans spend their attention on genuine judgment calls — is this analysis-population definition right for this trial? — and never on transcription, formatting, or reconciliation. The system is honest about what it could not do, which is precisely what makes the rest of its work trustworthy.

Accountability, in the end, is the point. Every regulated system must be able to answer “who decided this?” — and “the pipeline” is not an answer. The pipeline’s job is to make sure that question always has a name, a moment, and a record.

The test. Ask of any automated step: “When it is uncertain, what physically happens?” If uncertainty becomes a default value in an output, the system is quietly deciding things it was never asked to decide. If it becomes a routed question with a human name on the resolution, the loop is real.