To Train an AI Agent on Company Data, You Don't Actually Train Anything
The phrase "train an AI agent on company data" gets used constantly, and it sets nearly every business owner off in the wrong direction. People picture a machine learning project — engineers, GPUs, months of work, a budget that starts at $100,000. That's not what happens, and understanding why saves you a lot of money.
Modern AI agents already know how to read, write, and reason. What they don't know is your pricing, your service area, your warranty terms, and what your team tells a customer who calls asking about a 12-year-old furnace. You're not teaching the model to think. You're giving it a reference library and telling it when to look things up.
Fine-tuning vs. retrieval — and why retrieval wins
Fine-tuning bakes information into the model itself. It's expensive, slow to update, and when your prices change in March you have to redo it. Retrieval — technically RAG, retrieval-augmented generation — keeps your content in a searchable index the agent queries in real time. Change a document, and the agent's answer changes within minutes.
For 95% of growth-stage businesses, retrieval is the right answer. Fine-tuning only earns its cost when you need the agent to adopt a very specific writing style across thousands of outputs, or when you're working at a volume where shaving tokens actually moves the bill.
A fine-tuning project for a mid-size services company typically runs $30,000–$60,000 and needs to be redone every time your offerings change. A well-built retrieval setup on the same content usually costs $4,000–$12,000 and updates when you edit a document. Ask any vendor which one they're proposing before you sign.
Start With the 20 Questions Your Team Answers Every Week
Most companies begin by trying to dump everything into the agent — every folder, every PDF, ten years of email. It produces a slow, confused agent that confidently cites a policy you retired in 2023.
Work backwards instead. The agent exists to answer questions. So write down the questions first.
The one-afternoon source list
Sit down with whoever handles the phones and whoever handles email. Ask them to list the questions they answer more than twice a week. You will get 20–30 in about 45 minutes. Then, for each question, note where the correct answer currently lives — a document, a spreadsheet, someone's head, a page on your site.
That list is your scope. Anything not connected to a question on it doesn't go in version one. A dental practice we worked with started with 23 questions; 19 of them were answerable from four documents. The remaining four lived entirely in the office manager's memory, which turned into the most valuable 90 minutes of the whole project — we wrote them down.
What to Feed It — and What to Leave Out
Content quality matters far more than content volume. An agent with 40 clean, current documents will outperform one with 4,000 messy ones every single time.
The four buckets that actually matter
- Offerings and pricing: service descriptions, rate sheets, package tiers, what's included, what isn't. This is where most customer questions land.
- Policies and terms: warranty language, cancellation rules, service areas, payment terms, insurance or licensing details.
- Process and scheduling: what happens after someone books, typical lead times, what a customer should prepare, how emergencies are handled.
- Real answers to real questions: your best 30 email replies and support responses, lightly edited. This is the single most underused source, and it's the one that makes the agent sound like your company instead of a brochure.
What to keep out
Leave out anything you can't verify is current. Leave out internal strategy documents, HR files, financials, and anything containing customer PII unless you have a specific reason and the access controls to match. Leave out drafts. Leave out the 2019 pricing PDF that's still sitting in the shared drive — especially that one. An agent has no instinct for what's stale; if it's in the index, it's fair game for an answer.
An AI agent doesn't make your documentation better. It makes your documentation louder. Whatever's wrong in there is about to get repeated to every customer who asks.
Cleaning Your Content: The Part Nobody Warns You About
Budget real time here. On a typical project, content preparation is 60% of the work and the single biggest driver of whether the agent is any good.
The one-topic rule
Retrieval systems break documents into chunks and pull the most relevant ones. A 40-page operations manual covering fourteen unrelated topics chunks badly — the agent grabs a fragment with no context and answers half a question. Split it. One document, one topic, with a clear heading that says what the topic is.
Three other rules that pay for themselves:
- Kill the ambiguity. "Contact us for pricing" teaches the agent nothing. Give it the actual range and the actual conditions.
- Convert tables and images to text. A price list trapped in a scanned PDF or a JPG is invisible to the agent. Retype it.
- Date every document. Add "Last updated: [date]" at the top. It makes the 90-day review possible and gives you a way to spot rot.
Keeping It Accurate After Launch
The agent is right on day one and quietly wrong by month six. Not because the technology degrades — because your business changes and nobody updated the source document.
The 30-day accuracy audit
Every 30 days for the first quarter, then quarterly, do this: pull the last 100 conversations, sample 25, and have the person who actually knows the answers grade them right, wrong, or incomplete. Track the percentage. Anything below 90% means a source document needs fixing — and the log will tell you exactly which one.
Then close the loop operationally. When pricing changes, updating the source document goes on the same checklist as updating the website. If it isn't on someone's checklist, it won't happen.
Name one person as the content owner before launch. Not a committee, not "the team" — one name. The agents that stay accurate two years in all have this in common, and it costs nothing.
What It Costs and What It Returns
Real numbers from a services business with roughly $6M in revenue and four people fielding customer questions:
- Setup: ~$8,000, including content cleanup, index build, testing, and two rounds of tuning.
- Ongoing: ~$250/month in platform and usage costs, plus about two hours a month of content maintenance.
- Return: 61% of inbound questions resolved without a human. That's roughly 22 hours a week returned to a team whose loaded cost is $38/hour — about $43,000 a year.
Model it before you build it
Payback lands around month three, but only because the scope was tight and the content was clean. The projects that don't pay back are the ones that skipped the question list, indexed everything, and never named a content owner. If you want the full framework, we walk through it in our guide to calculating ROI on AI automation before you spend a dollar.
Start with the 20 questions. Everything else follows from there — and if you'd like a second set of eyes on which of your workflows is worth building first, book a strategy call and we'll map it with you.