Short answer
A small business needs four things to run AI in production: a model reached through an API, a workflow layer to connect systems, storage for what the workflow produces, and logging you actually read. No GPUs, no cluster, no vector database on day one. Add components only when a specific condition forces it — a data residency obligation, sustained volume that beats per-token pricing, or retrieval quality that has become the measurable bottleneck. Keep the model call behind one interface — on LYVIA's engagements that typically brings a provider switch down to about a day, a rule of thumb rather than a published benchmark — and monitor cost per run and output quality, not just uptime.
The four components you actually need
Reference architectures published by cloud vendors describe a company with a platform team and a budget line for it. Stripped down to what a company of ten to a hundred people needs to run real workflows, the list is short.
- A model, reached through an API. Whether that is a frontier proprietary model or a hosted open one is a separate decision, covered in our comparison of open-source and proprietary models.
- A workflow layer that triggers on events, calls the model, and writes results into the systems you already use. This is the piece that does the actual integration work.
- Somewhere to store what it produces — often a table in a database you already run, sometimes just the system of record itself. New storage is the exception, not the default.
- Logging you will read. Every run: input, decision, output, cost. Not a monitoring product — a table and the habit of looking at it during the first weeks.
That stack runs the majority of what small businesses automate. If you cannot point at which of the four a proposed component replaces or supports, it is probably premature.
What to deliberately not build yet
The most useful infrastructure decision at this scale is usually a decision not to build something. Each item below is genuinely valuable at a size most companies reading this have not reached.
- A vector database, before retrieval quality is a measured problem. Below a few thousand documents — an order of magnitude LYVIA sees across its own engagements, not a published benchmark — the search in tools you already run is usually enough.
- Your own GPU or a rented one running continuously, before volume is steady and high — the API-versus-self-hosted arithmetic is worked through in our comparison of open-source and proprietary models.
- A container orchestration setup for two workflows. Managed hosting costs less than the hours spent maintaining the alternative.
- A fine-tuned model, before better prompting and retrieval have been exhausted. Fine-tuning fixes tone and format far more often than it fixes knowledge.
- A full governance platform, before there are enough teams and workflows for governance to mean anything.
Every additional component is a thing that breaks quietly while nobody is watching, and a thing the next person has to understand. At this scale the maintenance burden — not the license cost — is what makes over-built stacks expensive.
The conditions that justify more
Rather than a maturity model, use conditions. Each of these is checkable, and each genuinely changes the right architecture.
- A written obligation that data must not leave your systems — from a client contract, a regulator or a tender commitment. This forces self-hosting and it is the only condition that does so unarguably.
- Sustained, predictable high volume where per-token pricing has clearly overtaken the fixed cost of hosting — that crossover is the subject of our open-source versus proprietary comparison — measured on real traffic rather than projected traffic.
- Retrieval quality as the measured bottleneck — you can show that the model answers badly because it received the wrong documents, not because the prompt was weak. That is when a proper vector store earns its place, and the underlying method is covered in our guide to retrieval over company documents.
- Latency that users feel in an interactive path, where a smaller or closer-hosted model changes the experience rather than the invoice.
The data layer decides more than the model does
The most common reason an AI project underperforms in a small company has nothing to do with the model. It is that the information the workflow needs is scattered across systems that do not talk to each other, exists in three inconsistent versions, or lives in someone's inbox.
No model fixes that. What helps is unglamorous: one system of record per kind of information, a consistent identifier so records can be matched, and access from the workflow layer without a manual export step. Companies that do this find their second and third automations dramatically cheaper than the first, because the plumbing already exists — which is also why the first project should be chosen partly on which plumbing it forces you to build.
Avoiding lock-in without over-engineering
Provider lock-in is a real risk and an easy one to over-correct. Building an abstraction layer that supports every provider equally costs more than the switch it protects against.
- One place in your code or workflow where the model is called — not provider-specific logic scattered through a dozen steps.
- Prompts and test cases in version control, so a switch can be validated by rerunning them rather than by hoping. That regression set is described in our guide to prompt engineering for business.
- Your own data in a format you control, exportable without the vendor's cooperation.
- Pinned model versions where the provider allows it, so an upgrade is a decision rather than an event.
The goal is not portability in principle. On LYVIA's engagements — a rule of thumb, not a published benchmark — it is being able to move within a week if pricing, terms or quality change, which is a much cheaper target to hit.
What to monitor once it is live
Standard uptime monitoring will tell you the workflow ran. It will not tell you the workflow is still doing its job, which is the failure mode that actually costs money here.
- Cost per run against forecast, alerting on drift rather than reviewing a monthly invoice after the fact.
- Escalation and refusal rate — how often the workflow hands back to a person. A sudden drop can mean it got better, or that it stopped noticing when it should ask.
- Latency as the waiting person experiences it, not average server time.
- A human reading a sample of real outputs on a schedule. This is the only check that catches quality decay, and the only one companies routinely skip.
Who runs this in a company with no platform team
Naming an owner at build time matters more than the architecture diagram. In practice the workable pattern in a company this size is one internal person who understands the process and can open the workflow tool, with an external partner for the parts that need engineering, and documentation short enough to actually be read — the four components, where each lives, and who to call.
What fails is infrastructure owned by nobody in particular: it works until the person who happened to build it changes role, and then it becomes a system nobody dares touch.
The order to add things in
Start with the four components against one real process. Get it into production and read the logs for a couple of weeks — an interval LYVIA uses on its own engagements, not a published benchmark. Add the second workflow on the same stack — that is where you find out whether your data layer is the problem. Only then consider anything from the "not yet" list, and only against a condition you can point at.
Sequenced that way, infrastructure follows the work instead of preceding it, which is also the argument behind the ordering in our AI implementation checklist.
Frequently asked questions
What infrastructure does a small business actually need to run AI?
Far less than the phrase suggests. For most companies of ten to a hundred people the stack is a model accessed through an API, a workflow tool to connect systems, somewhere to store what the workflow produces, and logging. No GPUs, no cluster, no platform team. Infrastructure only becomes a real project when data residency rules or sustained high volume force you to host the model yourself.
Do we need a vector database?
Only once retrieval over your own documents is genuinely part of the product, and often not even then at first. As a rule of thumb from LYVIA's own engagements — not a published benchmark — retrieval quality rarely becomes the bottleneck below a few thousand documents, and the search built into tools you already run is usually sufficient. Add a vector store when retrieval quality is the bottleneck you can measure, not in anticipation; our guide to retrieval over company documents covers how to tell.
Should we build on a cloud AI platform or assemble the pieces ourselves?
Assembling a few well-chosen pieces is usually cheaper and more portable at this scale. Full platforms make sense when you need governance features — audit trails, access control, model routing policies — across many teams. A company running three workflows is paying for governance it does not have the organization to use.
What is the most common infrastructure mistake?
Building for the volume you hope to have rather than the volume you have. It shows up as a self-hosted model idling, a Kubernetes cluster for two workflows, or a data warehouse feeding one dashboard. The cost is not only money — every component added is one more thing that breaks while nobody is watching.
How do we avoid getting locked into one model provider?
Keep the model call behind one internal interface rather than scattering provider-specific code through the workflow, keep prompts and test cases in version control, and store your own data in a format you control. On LYVIA's own engagements — a rule of thumb, not a published benchmark — swapping providers this way costs about a day plus a rerun of the test set. Absolute portability is not worth engineering for; being able to move within a week is the target we set with clients.
What needs monitoring once something is live?
Four things, and only one is technical uptime: cost per run against your forecast, the rate at which the workflow refuses or escalates, latency as experienced by the person waiting, and a sample of actual outputs read by a human on a schedule. A pipeline can be entirely green while quietly producing worse results, and only the last of those four catches it.
If you want the smallest stack that runs your first workflow in production — and a clear list of what not to build yet — that is where we start. Book a call.
