Short answer
AI automation is ordinary automation with one part changed: somewhere in the chain, a model makes a judgment instead of following a rule you wrote. That single change buys you unstructured input — emails, documents, calls — and costs you determinism: the same input can produce a different output, so you can no longer test the step once and trust it forever. Use rules wherever a rule is honest. Use a model where the rule would need thirty exceptions. And wherever you use one, replace the one-time test with a small, regular sample of real outputs.
The only distinction that matters
Most guides on this subject open with a list of tools. That is the wrong first question, because the tools cut across the distinction that actually governs the project. Nearly every automation you will build in a company of 10 to 100 people is a chain of steps, and each step is one of two things.
A deterministic step executes an instruction you wrote. Copy this field into that system. If the amount is over 5,000, route it to the director. Send the reminder on day 30. It gives the same answer every time, costs nothing per run, and when it is wrong, it is wrong in every case at once — which is why you find out fast.
A judgment step delegates a decision you would otherwise make with your own eyes. Which category does this incoming message belong to. Is this invoice a duplicate. What are the three obligations buried in this contract. Which of these applicants match the role. The instruction you give is a description of the goal, not a procedure — and the system works out the rest.
The handover is explicit in how these systems are built. Describing what happens when a model is given tools to work with, Anthropic's documentation puts it in one line: "Claude determines on each turn whether to call a tool or respond directly." (tool use overview, read August 2026). That sentence describes tool calling specifically — but the same shape holds for the simpler cases with no tools involved at all. When you ask a model to sort an email into one of five categories, you are not configuring a decision rule. You are handing the decision over and receiving its result.
Everything difficult about AI automation follows from that handover. Not from the model being unreliable — from the fact that a decision you delegated is a decision you are no longer watching.
Which side is your task on
Three questions, answered honestly about one specific step, settle it. They take a minute and they save the projects that would otherwise be built on the wrong assumption.
- Is the input structured? A form field, a database row, a number: structured. An email body, a PDF, a phone call, a photo of a delivery note: not. Unstructured input is the strongest single argument for a model, because writing rules against free text is how you end up with a hundred string comparisons that break on the first client who phrases it differently.
- Would two experienced colleagues agree on the answer? If yes, the rule exists and you have simply not written it down yet — write it down. If they would disagree, no model will resolve that for you either; you are looking at a definition problem wearing a technology costume.
- How many exceptions does the rule need? Two or three: keep the rule. Dozens, and growing every month as edge cases arrive: the rule has become an unmaintainable approximation of a judgment, and a model handles it better. There is no clean threshold here — the signal is the trend, not the count.
Notice that the middle question can disqualify the project entirely — and that is the most valuable outcome of the three. A process where nobody agrees on the correct answer cannot be automated in any form. It can only be decided.
What delegating judgment costs
The price is rarely the subscription. Three costs matter more, and only one of them appears on an invoice.
You lose repeatability. Run the same input through a deterministic step twice and you get the same result twice; that property is what makes a single test meaningful. A model-based step offers no such guarantee, and the practical consequence is not philosophical — it is that a run which worked on Tuesday proves very little about the same run on Friday. We hold ourselves to the same rule internally: one successful run is an anecdote, not evidence.
You lose visible failure. A broken deterministic step throws an error, and errors get noticed. A model-based step that has started getting things wrong returns a perfectly well-formed, confident, wrong answer. In a run log, those two look identical. This is the reason the verification section below is the substance of this article rather than an appendix.
You pay per decision. Rules are free to execute. Judgment is billed by the call, which changes the arithmetic at volume: a step that costs a fraction of a cent is irrelevant at fifty runs a day and is a line item at fifty thousand. The trap is not the price — it is discovering the price after the architecture assumed the step was free.
None of these are reasons to avoid model-based steps. They are reasons to spend them where they earn their keep, which usually means: a handful of steps inside an otherwise deterministic chain.
Checking something that does not repeat itself
Here is the part most implementations skip, and it is the part that decides whether the automation is still trustworthy in six months. You cannot test a judgment step the way you test a rule — one pass, green, done. You need an ongoing sample.
- Write down what a correct output is, before launch. Not a quality adjective — an example. Three good outputs and three bad ones, in a document. Without this, every later disagreement becomes an argument about taste, and the automation gets quietly abandoned instead of fixed.
- Sample a fixed number on a fixed schedule. Twenty outputs a week has been enough at the volumes we work with — a rule of thumb from our own engagements, not a published benchmark — and the fixed schedule matters more than the number: a review that happens when someone has time is a review that stops happening in the second month.
- Record disagreements, not just a count. Which case, what the system produced, what it should have produced. Ten logged disagreements tell you whether you have drift, a blind spot in one category, or a specification that was never clear. A percentage tells you none of that.
- Put a floor under the step. Where the output feeds an irreversible action, require a signal you can check mechanically — a value that must appear in a known list, an amount that must reconcile with a source document. The model proposes; something deterministic still holds the gate.
That last point is what makes hybrid systems the sane default. The model reads the contract and extracts the renewal date; a rule verifies that the date parses, falls in the future, and lies inside the contract term. Judgment where judgment is needed, arithmetic everywhere else.
If you build only one habit out of this article, build the sample. An unsampled judgment step is not automation you own — it is a colleague you never review, who never asks a question, and whose work you have stopped reading.
Where this lands in a company your size
In practice, in companies of 10 to 100 people, the judgment steps worth paying for cluster in three places, and they are all places where the input arrives as text somebody has to read.
Intake. Whatever lands in a shared inbox, a form, or a messaging channel and has to be classified, routed, or answered. This is the highest-volume unstructured input in most companies and the easiest to sample, because a wrong routing is visible and cheap.
Documents. Contracts, invoices, delivery notes, specifications — anything where the useful information exists but is buried in prose. Extraction is a judgment step; what you do with the extracted values is not, and should not be. That split is at its starkest with contract renewal dates, where a model reads the clause and arithmetic decides when to warn you. Answering questions across a whole corpus is a different architecture again — retrieval rather than extraction.
Drafting. First versions of recurring written work: replies, summaries, reports, proposals. Note that the value here is a draft a person edits, not a message that leaves the building unread — which is the same reversibility test as everywhere else.
What does not belong on this list: anything whose input is already structured. Reconciling numbers, moving records between systems, scheduling, reminders. Those are automation problems with settled solutions, and a model adds cost and uncertainty to both.
What to read next
This article deliberately stops at the line between rules and judgment. The rest of the decisions — which process to start with, how to run one in production, whether it paid for itself — each have their own logic, and we have written them separately rather than compressed them here.
- Deciding what to automate first — the selection method, and why the process you most want to automate is often the one to fix instead.
- Running automation without developers — failure handling, alerting and ownership: the half that decides whether it survives its first year.
- When a judgment step becomes an agent — what changes when the system chooses its own sequence of actions rather than one decision.
- Answering from your own documents — the specific architecture behind the document cases above.
- Measuring the return honestly — using hours that would genuinely have been reclaimed, not hours multiplied by a rate.
Frequently asked questions
What is the difference between automation and AI automation?
Ordinary automation executes rules you wrote. AI automation delegates a judgment you would otherwise make yourself — reading an email and deciding which category it belongs to, summarizing a document, choosing which of four templates fits a case. The distinction is not about which tools you buy; a single workflow usually contains both. It matters because the two have different failure modes. A rule that is wrong is wrong every time and you find it quickly. A judgment that is wrong is wrong occasionally, silently, and only in the cases nobody thought to check.
Do I need AI in my automation at all?
Often not, and that is a feature. If the decision inside your process can be written as a table of conditions, write the table. It is cheaper, it runs instantly, it costs nothing per execution, and it gives the same answer every time — which means you can test it once and trust it. Reach for a model when the input is unstructured (free text, a document, a recording) or when the rule would need dozens of exceptions to be correct. Paying per call for something an if-statement handles is the most common waste we see.
How do I know a model-based step is still working?
You sample it. A rule-based automation either runs or fails visibly; a model-based one degrades quietly, because a plausible wrong answer looks exactly like a right one in a run log. The practical method is to review a fixed number of outputs on a fixed schedule — twenty a week has been enough at the volumes we work with, which is our own rule of thumb rather than a published benchmark — and to record disagreements rather than just counting them. The disagreements tell you whether the step is drifting or whether your own definition of the right answer was never written down.
Should a model ever act without a human checking?
Yes, when the cost of a wrong output is lower than the cost of the review, and when the action is reversible. Categorizing an incoming message: let it run. Sending a reply to a client, issuing a credit note, changing a record in your accounting system: keep a person in the loop until you have real numbers on how often it is wrong. The mistake is treating this as a philosophical question rather than an arithmetic one — the answer changes per step, not per company.
If you would rather have the line drawn for you, on your own processes, that is the work we do. Book a call and bring the step you are least sure about.
