Research

The Wrong Channel

If you measure AI corrections from chat text, you are measuring the minority signal

By Merryl DMello, DMello Advisory · 3 min read

The claim: the standard way teams measure how much they correct their AI, reading the chat, captures the smaller channel and misses the dominant one. In one long-running corpus, the operator interrupted running actions 131 times against roughly 51 typed corrections, a ratio of 2.6 to 1. The two channels carry different information, so a correction metric built on text alone is not a slightly noisy proxy. It is a measure of a different behavior.

The evidence is a full-corpus extraction of interrupt events, 180 correction-relevant events in total hand-checked by the analyst, from 303 main sessions across six models.

The table that inverts

Interrupts per 100 prompts, by model, next to each model's verbal-correction rate:

The per-prompt interrupt ordering inverts the verbal-correction ordering. The model that attracted the fewest typed corrections drew the most interruptions. Any dashboard that ranks models or workflows by chat corrections would have scored this operator's best-working model relationship as the smoothest, when it was in fact the most actively steered.

Why the channels diverge

The transcripts show the split is functional, not emotional. Interrupts are pacing and control: stopping a tool call mid-flight, cutting off a question-list before it burns more context, halting a cost-consuming action during its explanation. Corrections are content: a wrong fact, a wrong assumption, a wrong file.

Two categories appear in the interrupt record that never once appear as typed text: irritation at verbosity, and stopping an action that was spending money. Nobody types "stop explaining." They hit the key. The chat record of those moments is empty by design, which is precisely why chat-derived metrics cannot see them.

There is also a per-model style effect that only the interrupt channel reveals. The model with heavy browser-automation sessions invites mid-flight redirection, so its interrupt rate is high while its correction rate is the lowest. The orchestration-style model delegates in long unattended stretches, leaving fewer interruptible moments, so it collects typed corrections instead. Same user, same months, opposite profiles depending on how the model works.

What this does to your metrics

Three concrete consequences for anyone building or buying AI-workflow analytics:

1. Correction rate is channel-dependent. A team that interrupts a lot will look "uncorrected" in chat text. A team that types corrections will look over-corrected. Neither number describes friction with the model.
2. Interruptions are an early signal worth counting. They fire mid-action, before a result exists to evaluate. A rising interrupt rate around a specific workflow is a leading indicator that the workflow is spending time or money wrongly.
3. The measurement is nearly free. Interrupt events are structural markers in the transcript stream, not semantics to interpret. Counting them is a script, not a project. The corpus analysis hand-checked 180 events to validate the classifier; after that, counting was mechanical.

What would change my mind

• A corpus where typed corrections outnumber interrupts for a high-volume operator. In this corpus the aggregate ran 131 interrupts to roughly 51 corrections; per-model correction counts were not extracted, so a per-model flip is untested.
• Evidence that chat-derived correction metrics predict outcomes (rework, cost, satisfaction) better than interrupt-inclusive ones. No published comparison exists to my knowledge, which is itself the finding.
• A workflow tool that already surfaces interrupt rates per model or per workflow. I can find none; the event is in every transcript and in no dashboard.

The author runs dmello-advisory and instruments AI workflows for small and mid-size companies, measuring the channels that matter instead of the ones that are easy. Eighteen years building and leading products at Intel, HP, Rackspace, BioNTech, and Phoenix Contact. dmello-advisory.com.

All research papers

Last updated: 16 October 2026