The sign-off problem nobody warns you about
A rules-based automation can be signed off the way you sign off a spreadsheet formula. Feed it the same input twice and you get the same output twice. You read the logic, you reason about it, you accept it. That habit is what makes the first AI workflow in a small company so dangerous: it looks like the same object, it sits on the same canvas, and it invites the same casual sign-off.
The n8n documentation puts the difference plainly: AI models are fundamentally different from code, because code is deterministic and can be reasoned about, while models are black boxes whose output you can only measure by running data through them and observing what comes back. Confidence, in that framing, is not something you derive from reading the workflow. It is something you accumulate by running enough inputs that reflect the edge cases production will actually hand you.
That reframing has a practical consequence for a 10 to 100 person company. You cannot budget one afternoon of "testing" at the end of a build and call it verification. You need staged testing: a sequence where each stage costs a little more effort than the last and each stage is allowed to stop the launch. This article sets out the sequence LYVIA recommends, the methods available at each stage, and — because this matters more than the methods — what each one does not prove.
Everything below describes a proposed working method and the documented behaviour of the tooling we use. It is not a claim about outcomes at any particular company, and none of it makes an AI workflow reliable by itself.
Stage 0: stop touching live records
The first mistake is not a bad prompt. It is testing against production data because that is what was easiest to wire up. Every test run then becomes a real API call, a real row written somewhere, a real customer record read by a system nobody has reviewed yet — and, if the workflow sends anything, a real message with your name on it.
n8n separates two features that together solve this. Data mocking means creating or simulating test data without connecting to a real source: a custom dataset built in the Code node, a handful of fields set in the Edit Fields (Set) node, or the sample dataset the Customer Datastore node returns. Data pinning means saving a node's output and reusing that saved data on future executions instead of fetching fresh data. The docs recommend combining them — mock, edit the values to create the edge case you care about, then pin the result so every subsequent run sees the same input.
The gains are the obvious ones and one less obvious one. You stop consuming rate limits and usage quotas on an external system you are only poking at. You stop depending on an external trigger firing to test a webhook-driven flow. And you get a consistent dataset, which means that when the output changes between two runs you know the change came from your edit and not from the data moving underneath you.
The limits are documented and worth respecting. Pinning is a development feature: it is not available for production workflow executions, and neither is editing pinned data. You can only pin data on nodes with a single main output, and you cannot pin output that includes binary data — which rules out the exact case many small businesses start with, a scanned invoice or a photo. For those, plan on a small folder of fixture files you control instead.
The rule we apply: before a single live record enters the workflow, the happy path and at least three ugly paths must run end to end on mocked or pinned input. Ugly means empty field, wrong language, and something the process was never designed to receive.
Stage 1: a quick evaluation on a handful of realistic cases
Once the workflow runs on invented input, the question becomes whether it does the right thing on input that looks like yours. This is where a light evaluation earns its place. The idea is small on purpose: a test dataset of a handful of examples, run through the workflow one at a time, with the outputs written back next to the inputs so you can compare them side by side.
Mechanically, in n8n, that means four steps. Create a dataset — a Data Table or a Google Sheet, one row per test case, with columns for the input, optionally the expected output, and blank columns for the actual output. Add an Evaluation trigger, which emits one item per row; while you are wiring it up you can cap it at one row or execute just that node, then use "Evaluate all" to run the whole set in sequence. Add the Evaluation node's Set outputs operation at a point after the workflow has produced what you want to inspect, and map those values into the right columns. Then run it and read the results.
The docs are candid that at this stage expected outputs are optional and a formal metric is optional too, because building a clean, comprehensive dataset is genuinely hard and a handful of examples is often enough to iterate a build to a releasable state. That is the right trade for a first version. It is also where the big quality jumps happen: light evaluation is characterised as producing large improvements per iteration, on a small dataset, precisely because you are still fixing structural mistakes rather than shaving points off a score.
Two honest limits. Eyeballing ten outputs tells you nothing about the distribution of failures — you are sampling your own imagination, not production. And the moment you have more rows than you are willing to read carefully, this method quietly stops working; you will skim, agree with yourself, and ship. Notice that moment when it arrives.
One access note, because it changes what you can plan: light evaluations are documented as available on all n8n Cloud plans and, for self-hosted, on Registered Community, Business and Enterprise. Feature tiers move, so confirm against the current docs before you design a process around them.
Where the sample data comes from, without exposing customers
A realistic dataset and a safe dataset are not in conflict, but you have to build it deliberately. Three sources, in the order we prefer them.
Hand-written cases come first, and they should be written by whoever does the work today, not by whoever is building the workflow. Ask them for the five requests that are always misfiled, the two that arrive in the wrong format, and the one that everyone escalates. That single conversation usually surfaces edge cases no prompt author would invent.
Redacted real cases come second. Take the structure and the awkwardness of a real ticket, contract clause or invoice line, then replace names, account numbers, addresses and anything else that identifies a person or a deal. The point of the test case is the shape of the input, not whose input it was. Keep the dataset itself in scope of your normal data rules: if a Google Sheet holding it is shared with the whole company, you have created a new exposure, not removed one.
Synthetic and AI-generated cases come third, and the docs list them as a legitimate dataset source at both stages. They are useful for volume and for combinatorial variation. They are also biased toward what a model thinks your business looks like, which is exactly the blind spot you are trying to cover. Treat them as padding around a core of human-written cases, never as the core.
Later, when the workflow is live, the best source appears: the production execution that went wrong. The documented discipline is to add that failing input to the dataset and re-run the whole dataset after fixing it, as a regression test, to confirm the fix did not quietly break something else. That habit is the single highest-value testing practice in this article, and it costs nothing but the decision to do it every time.
Stage 2: metrics, once individual reading stops scaling
Past a certain dataset size you cannot form a view by reading. You need a number per test case that rolls up to a number per run, so you can compare this run to the last one. That is metric-based evaluation, and n8n's version builds directly on the light setup: same dataset, same trigger, same Set outputs, plus logic that scores the result and a Set Metrics operation that records it.
The built-in metrics are worth understanding individually, because each measures something narrow. Correctness (AI-based) asks whether the answer's meaning is consistent with a supplied reference answer, scored 1 to 5. Helpfulness (AI-based) asks whether the response answers the query, also 1 to 5. String Similarity measures character-by-character edit distance from a reference, returning 0 to 1. Categorization returns 1 for an exact match with the reference and 0 otherwise. Tools Used returns 0 to 1 for whether the execution used tools. Custom metrics let you compute your own number in the workflow and map it in — the documented example being RAG document relevance, whether the documents a vector store retrieved were actually relevant to the question.
Choosing among them is mostly about whether a single correct answer exists. Classification, extraction and routing have one, so Categorization and String Similarity do real work and cost nothing beyond compute. Drafted replies and summaries do not: two very different answers can both be right, which is where an LLM-as-judge metric like Correctness or Helpfulness becomes the only practical option. A useful pattern from the same material: run the cheap, fast model in production and the slower, stronger one as the judge over a small set of question and answer pairs.
Three cautions. AI-based metrics inherit the non-determinism of the thing they are judging, so a 3.8 versus a 3.9 is not necessarily a change. An average can improve while the one input that matters most to your business regresses, which is why the docs recommend reading both the aggregate and the individual cases. And String Similarity rewards resemblance, not truth — a confidently wrong answer phrased like the reference scores well.
Two operational notes. Metrics add latency and cost, so put the scoring logic behind the Check if evaluating operation and keep it out of production executions. And running cases in parallel speeds things up: the documented default maximum is 1 for self-hosted Community and Cloud Pro, 3 for self-hosted Business, 5 for Cloud and self-hosted Enterprise, with self-hosted instances able to override via N8N_CONCURRENCY_EVALUATION_LIMIT — at the cost of a higher chance of hitting upstream LLM rate limits. Metric-based evaluation is itself documented as a Cloud Pro and Enterprise, or self-hosted Enterprise, feature, with Registered Community and Starter users able to use it for a single workflow.
A baseline, or the numbers mean nothing
A score on its own is not information. A score compared to the same dataset's previous score is. Before you change anything, run the current version against the fixed dataset and keep the outputs and the metrics. That saved run is your baseline, and every subsequent prompt edit, model swap or retrieval change has something concrete to beat.
This is what makes prompt versioning useful rather than decorative: each version is attached to evaluation results across the dataset instead of to a memory of a few good-looking replies. It matters more for agents than for single-shot prompts, because a wording change there can alter which tools get called and in what order, not just the phrasing of the final answer.
The failure mode a baseline catches is the quiet one. An obvious regression shows up in a side-by-side comparison. A slow drift — average correctness sliding while every individual output still reads fine — only shows up as a trend across runs. If you look at nothing else after launch, look at whether the trend line is going the wrong way.
When a score drops and you cannot see why, you need the trace rather than the summary. For self-hosted n8n instances, a LangSmith integration adds tracing for LangChain-based workflows so you can inspect spans inside an execution; that tracing is documented as self-hosted only, not available on n8n Cloud. Plenty of teams stay with observability platforms or code-first frameworks instead — Promptfoo, DeepEval, LangSmith, Braintrust, Langfuse, Arize Phoenix all occupy that space. The trade is straightforward: evaluation that lives in your CI pipeline is more rigorous and more work to maintain; evaluation that lives on the same canvas as the workflow is easier to keep alive in a small team.
Stage 3: a phased rollout, because tests are not production
Every stage so far ran on data you chose. Production will hand the workflow inputs you did not imagine, at volumes you did not simulate, from people who have no idea a model is involved. So the launch itself should be staged, and each phase should have a stated exit condition written before it starts.
Shadow phase. The workflow runs on real inputs and produces real outputs, but nothing it produces reaches a customer or a system of record. A person does the work as usual; the workflow's version sits next to theirs. You are comparing two answers on live inputs and discovering the input shapes your dataset missed. Exit condition: a fixed number of consecutive cases where the reviewer would have accepted the output without material edits.
Narrow phase. The workflow acts, but inside a boundary you can absorb — one queue, one client segment, one document type, low-value cases only — and with human approval on anything that leaves the company. Exit condition: the approval reviewer stops finding changes worth making, and the failure types you do see are ones you understood in advance.
Wide phase. The boundary widens and the approval gate narrows to the cases that actually warrant it: irreversible actions, money, anything a customer sees for the first time. This phase never ends. It is the steady state, which means somebody owns it — a named person who reads the metric trend, adds new failing inputs to the dataset, and re-runs the regression set after every change.
Rollback is not a phase, it is a precondition. Before phase two, write down the two sentences that answer: what turns this off, and who may decide. In practice that usually means a toggle that reverts the step to the manual process, a queue that holds items instead of dropping them while it is off, and a way to identify and correct anything the workflow already touched. If you cannot write those sentences, you are not ready to let the workflow act — and this rollback design is our recommended practice, not something the tooling provides for you.
What staged testing does not give you
It does not give you a guarantee. Running a dataset builds confidence that the workflow handled those inputs that time; it does not bound behaviour on inputs you have never seen. Anyone selling an evaluation setup as reliability is selling something the method cannot deliver.
It does not cover changes you did not make. The model behind an API can be updated, deprecated or re-tuned without your involvement, and a retrieval corpus drifts as documents are added. This is an argument for re-running the regression set on a schedule rather than only on your own edits, and for keeping the dataset small enough that doing so stays cheap.
It does not measure what you did not encode. If your dataset contains no cases in a second language, no cases from your most difficult client, and no cases where the input is simply wrong, your metrics will look healthy while those three categories fail in production. A test suite is a statement about what you decided to care about.
And it does not settle several questions that we consider genuinely open. How large a dataset is enough for a specific workflow — nobody can answer that from first principles; it depends on how varied your inputs are and how expensive a mistake is. How much to trust an LLM judge versus a human reviewer, and how often to re-check the judge against human ratings. Whether a metric that improves on average but worsens on your hardest cases represents progress. How to evaluate a multi-step agent when the final answer is acceptable but the path it took was not. We treat these as things to decide explicitly per workflow and revisit, not as solved problems.
A one-page plan you can fill in this week
Write down the workflow's job in one sentence, including what it is not allowed to do. Then list the inputs it will receive, and ask the person who handles them today which ones are awkward.
Build twelve to twenty test cases: hand-written awkward ones first, redacted real ones second, synthetic variations last. Put them in a table with an input column, an expected output column where a single right answer exists, and blank actual-output columns.
Mock and pin everything external so the workflow can run end to end without calling a live system, remembering that pinning is a development-only feature and does not cover binary data.
Run a light evaluation, read every output next to its expected value, and fix structural problems. Only when reading stops scaling, add one or two metrics that match the task — exact-match style metrics for classification and extraction, an AI-based judge for drafted text — and save the first run as your baseline.
Then stage the launch: shadow, narrow with approval, wide with a narrowed gate. Name the owner. Write the rollback sentences. Re-run the whole dataset after every change, and add every production failure to it.
If you would rather have this designed and built with your own processes and data rules in the room, that is the kind of engagement LYVIA scopes after a discovery call at book.lyv-ia.com. Scope and quote come out of that conversation; we do not publish prices, and we do not promise a reliability number before seeing the workflow.
FAQ
How many test cases do we need before launching an AI workflow?
There is no defensible universal number, and we will not invent one. The documented distinction is between a small hand-selected set used while building — often just a handful of examples, enough to iterate to a releasable state — and a larger dataset assembled later from production executions, where you need metrics because you cannot read every result. Practically, we aim for enough cases that every input shape the process actually receives appears at least once, including the awkward ones the current handler complains about. Then the dataset grows one case at a time: every production failure becomes a permanent test case, and the whole set is re-run as a regression check after each fix.
Can we just test on real customer data? It is more realistic.
It is more realistic and it is the wrong default. Testing on live records means every trial run reads real personal data through a system nobody has reviewed yet, consumes real API quota, and risks a real message going out. The safer pattern is to mock the input — build it in a Code or Set-style node, or use a sample dataset node — then pin that output so the same values are reused on later runs instead of fetching fresh data. When you do want the realism of real cases, redact them first: keep the structure and the messiness, remove names, account numbers and anything identifying. Note that in n8n, pinning and editing pinned data are development features, not available for production executions, and pinning does not work for binary output such as scanned files.
Which quality metric should we track?
It depends on whether one correct answer exists. For classification, extraction and routing, an exact-match metric (Categorization, returning 1 or 0) or a character-level similarity score works and is cheap to compute. For drafted replies, summaries and anything where two different answers can both be good, you need an AI-based judge — n8n ships Correctness and Helpfulness, both scored 1 to 5 against a reference or the original query — or a custom metric you compute yourself, such as whether retrieved documents were relevant in a RAG workflow. Track at most two or three, always against a saved baseline run, and read the individual cases as well as the average: an improving average can hide a regression on the input that matters most to you.
Does a passing evaluation mean the workflow is reliable?
No. An evaluation tells you how the workflow behaved on the inputs you chose, at the time you ran it. It says nothing about inputs you never imagined, and it cannot account for changes you did not make — a provider updating the model behind an API, or a document corpus drifting as files are added. That is why the method here ends with a phased rollout and a rollback plan rather than with a green score: shadow running on real inputs, then a narrow live boundary with human approval, then a wider deployment with the approval gate kept on irreversible or customer-facing actions. Reliability is an ongoing operational commitment with a named owner, not a test result.
Do we need a separate evaluation framework, or can testing live in the workflow tool?
Both approaches are legitimate and the trade-off is maintenance versus rigour. Code-first frameworks and observability platforms — Promptfoo, DeepEval, LangSmith, Braintrust, Langfuse, Arize Phoenix are the commonly cited options — put evaluation next to your code and CI pipeline, which suits teams with engineering capacity to keep a second system alive. Running evaluations inside the automation platform keeps the test dataset, the outputs and the run comparison on the same canvas as the workflow, which is usually what survives in a 10 to 100 person company. Check feature availability before you plan around it: n8n documents light evaluations on all Cloud plans and on Registered Community, Business and Enterprise self-hosted, while metric-based evaluation is a Cloud Pro/Enterprise or self-hosted Enterprise feature, with Registered Community and Starter limited to a single workflow. Tiers change, so verify against the current documentation.