Why the Questions You Ask an AI Agency Before Hiring Matter More Than the Demo
If you're compiling questions to ask an AI agency before hiring one, you're already ahead of most buyers. The typical evaluation goes like this: the agency shows a slick demo, drops a few client logos, quotes a number, and the deal gets decided on rapport. Six months later the "AI transformation" is a chatbot nobody uses and a $40,000 invoice nobody wants to explain to the board.
Industry surveys consistently put AI project failure rates around 70–80%. The pattern behind those failures is rarely bad technology — it's projects that were never tied to a business outcome in the first place. The demo can't tell you whether an agency will avoid that trap. The right questions can.
What a demo hides
A demo shows the agency's best 10 minutes under perfect conditions. It doesn't show how they scope, how they price, what happens when your messy CRM data breaks their workflow, or who owns the system when the contract ends. Every question below is designed to expose one of those hidden variables before you sign, not after.
Questions 1–3: ROI and the Business Case
Start here. If an agency stumbles on these three, the rest of the conversation is academic.
1. "What's the projected ROI on this project, and how did you calculate it?"
A serious firm builds the return model before the build starts: hours saved × loaded labor cost, leads recovered × close rate × average deal value, or missed calls captured × booking rate. If the answer is "AI will transform your operations" instead of a number with assumptions you can challenge, walk. This is the entire premise of an ROI-first AI strategy — no build begins until the math works.
2. "What would make you tell us NOT to do this project?"
Agencies that only say yes are order-takers, not advisors. A good partner can name disqualifiers: data too fragmented, volume too low to justify the build, a $200/month off-the-shelf tool that solves 80% of the problem. If they can't describe a project they'd turn down, every project looks fundable — with your money.
3. "Which of your past projects failed, and why?"
Anyone who's shipped real AI systems has scars — an integration that fought back, an adoption problem, a model that needed three rounds of tuning. You're not disqualifying them for failure; you're checking whether they learn from it. "We've never had a failed project" is either a very short track record or a very flexible definition of success.
Ask the agency to complete this sentence: "This project costs $X and returns $Y per month, breaking even in Z months." If they can't fill in the variables — or the payback period is longer than 12 months for a mid-five-figure project — the business case isn't there yet.
Questions 4–6: Ownership, Data, and Lock-In
These are the questions buyers skip and then regret in year two.
4. "If we part ways, what do we keep?"
Get specific: Who owns the prompts, workflows, and automation logic? Are the accounts (CRM integrations, phone numbers, API keys) in your name or theirs? A voice AI phone number that lives in the agency's account is a hostage, not an asset. The right answer: everything runs in accounts you control, and offboarding is documented before onboarding starts.
5. "Where does our data go, and who can see it?"
Your customer records, call recordings, and financials will flow through this system. Ask which AI providers touch the data, whether it's used for model training, and how access is controlled. Critical for dental and legal practices with HIPAA or privilege obligations — an agency that shrugs at this question has never worked in a regulated industry.
6. "What happens when the underlying AI models change?"
Models get deprecated and APIs change — several times a year. Ask who monitors the system, who pays for updates, and whether maintenance is included or billed hourly. A system with no maintenance plan is a system with an expiration date.
Questions 7–8: Process and Timeline
How an agency runs a project tells you more than what they promise to build.
7. "Walk me through your first 30 days."
The right answer starts with discovery — mapping your workflows, auditing your systems, quantifying the opportunity — not with building. A firm that starts coding in week one is building what they assumed, not what you need. A realistic sequence looks like the 90-day implementation roadmap: audit, pilot, deploy, measure.
8. "Who actually does the work?"
The person pitching you is often not the person building your system. Ask who your day-to-day contact is, whether work is done in-house or subcontracted, and how many active clients each delivery person carries. You want operators who understand business workflows — not just engineers who understand APIs.
Hiring an AI agency is not a technology decision. It's a hiring decision. You're adding a team member who happens to build software — interview them like one.
Questions 9–10: Pricing and Accountability
The last two questions protect you after the contract is signed.
9. "What's the all-in monthly cost after launch?"
The build fee is only part of the number. Get the recurring costs in writing: AI API usage, platform subscriptions, telephony minutes, maintenance retainer. A $15,000 build with $1,800/month of unmentioned run-rate costs is a very different investment than the one you were pitched. Push for a total-cost-of-ownership number over 24 months.
10. "What metric will we review in 90 days, and what happens if we miss it?"
This is the accountability question. A confident agency commits to a measurable target — response time under 5 minutes, 90% of calls answered, 10 hours/week of admin time recovered — and a defined review date. Vague success criteria ("improved efficiency") are how bad projects survive long enough to get expensive. If they won't name a number, they don't believe the project works.
How to Score the Answers
You don't need perfect answers to all 10 — you need honest ones. Score each answer as green (specific, quantified, documented), yellow (plausible but vague), or red (deflection, jargon, or "trust us"). Two or more reds, especially on ROI or ownership, should end the conversation.
Red flags that override everything else
No ROI model before the build. No documented offboarding. Guaranteed outcomes with no discovery phase. Pressure to sign an annual contract before a pilot. Any one of these has predicted more failed AI projects than any technology choice ever has — a pattern we break down in why 70% of AI projects fail.
Send questions 1, 4, 9, and 10 to the agency by email before the sales call. The firms worth hiring will answer in writing without hesitation. The ones that insist on "hopping on a call first" to answer basic ROI and ownership questions have just answered them.
We built Blake Agency to pass this exact interview. Every engagement starts with the ROI model, every system runs in accounts you own, and every project ships with a 90-day metric we're accountable to. If you want to see what those answers look like in practice, book a free strategy call — bring all 10 questions.