The purchase order workflow on my bench asks a model one question about every supplier reply. Is it a clean acknowledgment, a date mismatch, a price mismatch, a quantity mismatch, or unreadable? Five answers. The model is a large language model (LLM), the ChatGPT kind, so it answers by writing: a decision, a confidence between 0 and 1, and a sentence or two of rationale, all in a set format.
The first version of that prompt asked for one format. The step downstream that reads the answer expected another. Every time the model did exactly what the prompt said, that step reported an empty response, and once its retries ran out the case went down the path for infrastructure failures. For days it looked like a flaky model.
The confidence number has its own problem. The file that sets the workflow’s review thresholds carries a warning: “The number is a token, not a probability.” The model writes that number the way it writes any other word. Nothing measured it. Miao Xiong and colleagues, in a study presented at ICLR 2024, a major machine-learning conference, found that LLMs, when they state their confidence, tend to be overconfident.
For a question like that one, a classifier gives you more control than an LLM does.
01What a Classifier Gives You
A classifier is a model that can only answer from a list you define. It doesn’t write its answer. It gives you a probability for each option on the list.
A malformed answer can’t happen, because there’s no format to get wrong. That’s poka-yoke.
And the probability is something you can set a threshold on. Above it, the case moves. Below it, a buyer looks. That threshold is a control you own and can audit, provided the number means what it says.
One kind of classifier is new. A decision model comes pre-trained to pick from any list you hand it, so it works without your history. TypeSafe AI put its decision model, Jev, into early access on September 15. Cloudflare released two on October 1 and published the models themselves on Hugging Face under an Apache 2.0 license, so they’ll run on your own hardware (which matters for the reasons in last month’s article). Microsoft has one too, Microsoft-Decision-1, which InfoWorld described on October 13 as “an AI model that returns probabilities for predefined answers.”
02Your History Is the Training Set
In late September two researchers at Texas State University, Amir Rafe and Subasish Das, tested Jev and seven open decision models against older classifiers and general LLMs, all under one setup. Their paper is a preprint and hasn’t been peer reviewed.
On a test with 150 categories of requests, a logistic regression, the textbook classifier, run on top of a small open model that turns text into numbers and trained on labeled examples, scored 94.8%. Jev scored 91.3%. On a second, synthetic test, Jev scored 73.2% to its 70.2%, a gap the authors didn’t find significant. On the 150-category test the logistic regression was also the best calibrated model, meaning its probabilities came closest to how often it was actually right. In the authors’ words, “a small classifier trained on that traffic was more accurate and better calibrated on the intent benchmark.”
Where there weren’t any labeled examples, it mostly went the other way. On the two main tests, six of the eight decision models beat the one classifier built to work without training. It wasn’t universal. On a test of reading emotion in text, that classifier beat them.
A trained classifier needs history, meaning cases where somebody already made the call. Every supplier reply a buyer has already worked is one, if the ERP kept what the buyer did with it. That history is the training set.
03Where the Classifier Falls Short
A classifier can only answer from the list, and that’s also its weak spot. Hand it a reply that fits none of the answers and it’ll still pick one.
Rafe and Das measured this. They set Jev’s threshold so that about 5 in 100 of the answers it gave on real requests were wrong. At that setting it still accepted 31% of requests that belonged to no category at all. Adding an explicit “none of these” option cut that to 19.5%. Two of the open models held it to 3% or less.
Wording matters. Each option on the list has a short label and a description. When the researchers swapped the labels yes and no on two-option questions, leaving the descriptions unchanged, about half of Jev’s answers flipped. Some of the open models barely moved, and one did worse. Neutral labels, with the meaning carried by the description, fixed most of it for Jev.
A threshold only becomes a control after you test it.
The paper says it “has to be set for one model on one task on held-out labeled data.” That’s the plant’s own history again.
My workflow’s LLM writes a sentence or two saying which fields it compared. A classifier returns a probability and nothing else. The person approving a chase email needs the reason, so with a classifier the reason has to be built from the PO fields and the rule that fired. I haven’t put a classifier on that workflow yet, so I can’t tell you it wins there.
04What the LLM Still Does
Some steps don’t have their answer on a list. Once a reply is classified as a date mismatch, the new promise date has to be pulled out of the email and checked against the PO. If the supplier needs a chase, somebody has to write it. That’s LLM work, with a person approving anything that leaves the building.
So split the work by the question each step asks. If the answer’s on a list and you’ve got history, train a classifier on it. If it’s on a list and you don’t, use a decision model. If the answer has to be written, use an LLM. Whichever one runs, a rule or a person still decides before anything irreversible happens.
Take the last AI workflow somebody pitched you and walk it step by step. How many of those answers were already on a list?