- Accuracy has no single definition. Split it into four things you can score separately: right answer, right source, right permissions and a correct refusal when no answer exists.
- Write your question set first. A vendor-built demo set proves the product works on the vendor's data. Yours should come from real questions your people ask, with answers your experts have signed off on.
- Five tests catch most of what breaks later: stale documents, permissions, no-answer cases, conflicting sources, and a Monday morning test with real users on real work.
- Measure consistency, not just peak performance. Run every question several times. A system that is right once and wrong the next time is not ready for a process.
- Pilots hide production problems. Connector sync, permission drift, scale and ownership are the things that fail after the pilot is declared a success.
Most enterprise AI pilots do not fail loudly. They end with a slide that says the demo went well, a few enthusiastic users, and a decision to expand to more teams. Then somebody in finance asks the assistant about a contract term, gets an answer from a document that was replaced last spring, and the rollout quietly stalls.
The pattern is common enough to have numbers attached. Gartner projects that more than 40% of agentic AI projects will be cancelled by the end of 2027, citing cost, unclear value and weak risk controls. MIT's NANDA initiative found that roughly 95% of enterprise generative AI pilots showed no measurable P&L impact. Neither study blames the models. Both point at the work around them, and testing is the biggest piece of that work.
This post is about that work. It covers how to define accuracy so you can measure it, how to build a question set that is yours, five tests worth running in every pilot, what to track, what usually breaks between pilot and production, and what to ask a vendor before you sign. It is the testing counterpart to our guide on deploying AI without rebuilding your existing systems: that one is about where the intelligence sits, this one is about proving it works before a thousand people depend on it.
Part 1 · Definitions
What does it actually mean for an AI system to be accurate?
One number hides four different ways of being wrong, and each has a different owner and a different fix.
When someone says an AI system is 95% accurate, ask them what was counted. The number usually means that a reviewer looked at an answer and thought it was good. That is a useful signal, but it hides at least four separate failure modes.
Enterprise AI accuracy, defined
Right answer. The response is factually correct and complete for the question asked. This is the one everybody tests.
Right source. The answer came from the document it should have come from: the current policy, the signed contract, the approved version. A correct answer pulled from the wrong document is luck, and luck does not scale.
Right permissions. The answer used only material the person asking is allowed to see. A system can be perfectly accurate and still be a data leak. If that risk is new to your committee, our explainer on where your data actually goes when you use enterprise AI covers the architecture behind it.
Right refusal. When the answer does not exist in your data, or the sources disagree, the system says so instead of inventing something fluent. This is the one most pilots skip, and it is the one that causes the worst incidents.
Score these four separately and you will find that accuracy splits into very different numbers. A system might be 92% right on answers, 80% right on sources, and 40% right on refusals. A single blended score would have told you 85% and sent you to production.
A correct answer from the wrong document is luck, and luck does not scale.
Eerly AI StudioThere is also a consistency question that blended scores hide. Researchers at Sierra built a benchmark called tau-bench around it, using a stricter measure they call pass^k: did the agent get the same task right every time across k attempts? The gap is large. By the benchmark's own rule of thumb, an agent that is right 90% of the time on a single run gets through eight repeats without a miss only about 43% of the time. Your business does not ask whether the system worked at least once. It asks whether it works every time.
Part 2 · The question set
Build your own question set before the vendor builds theirs
A curated demo set proves the product works on questions chosen by someone who already knows what it handles well.
Vendors will offer to run a pilot on a curated set of questions. That is reasonable for a demo and a poor basis for a decision. Build yours first, and keep it private until the pilot starts.
A workable question set has a few properties.
- It comes from real work. Ask ten to fifteen people across finance, HR, operations, legal and IT for the last questions they actually looked up or asked a colleague. Include the badly worded ones. Real users do not write clean prompts.
- It has expert-approved answers. For each question, a named person writes the correct answer and points to the source document. If two experts disagree, that question has just told you something about your data, and it belongs in the conflict test below.
- It includes questions that should fail. At least a fifth of the set should be questions where the right behaviour is to decline: topics not in your data, questions a particular role should not be able to answer, and questions that depend on a policy that has been retired.
- It is big enough to mean something. As a rule of thumb, 100 to 200 questions is enough to see patterns by department and question type. Fewer than 50 and a handful of lucky answers can swing the result.
- It is versioned. Keep it in a shared file with owners and dates. You will rerun it every time the vendor changes a model, a connector or a setting, and it will become your regression suite.
One more rule: never tune the system against the test set and then report results on the same set. If the vendor sees your questions, hold back a portion they have not seen, and treat that portion as the real score.
Part 3 · The five tests
The five tests every enterprise AI pilot should pass
Each one targets a specific way enterprise AI systems fail once they meet real company data.
Run these in addition to your general question set. They are cheap to set up and they surface problems that averages smooth over.
Whether the system prefers current information over outdated information.
Pick ten topics where a policy, price list or procedure has changed in the last two years. Keep the old version in the connected data, as it would be in real life, and ask questions where the old and new versions give different answers. Examples: travel limits, notice periods, approval thresholds, a vendor's payment terms.
The current version is used, and the answer names it, with a date or version where possible.
A confident answer from the superseded document, with no hint that a newer one exists. This is the most common failure in pilots, because search systems rank by similarity, not by freshness.
Whether the system respects who is asking.
Create or borrow test accounts at three levels: a general employee, a department manager and a senior administrator. Ask each of them the same set of sensitive questions: salary bands, board papers, M&A files, disciplinary records, customer contracts. Then try indirect routes. Ask for a summary of everything about project X, or ask a question whose answer can be inferred from a document the user cannot open.
Each account sees exactly what it would see in the source system and nothing more. Restricted material is neither quoted nor paraphrased, and the system does not confirm that it exists.
Any leak at all. One leak is enough to stop the pilot. Then test revocation: remove access for a test user in the source system and see how long the AI system takes to follow. Do not take it on faith.
Whether the system knows what it does not know.
Ask questions that sound reasonable but have no answer in your data: the notice period for a contract type you do not use, the policy for an office you do not have, a figure for a quarter that has not closed. Add a few questions that are close to real topics but missing a key detail.
A clear statement that the answer could not be found, ideally with a pointer to where it might live or who owns the topic. Partial answers are labelled as partial.
A fluent, plausible and wrong answer. This is the failure mode engineers sometimes call fail-plausible, where an error shows up not as silence or a crash but as confident prose. It is the hardest failure for a user to notice, which is why you test for it on purpose.
What happens when your own sources disagree.
Every company has them: the handbook says 20 days and the HR system says 18, the contract template differs from the signed version, two regional policies contradict each other. Collect ten real conflicts and ask questions that sit right on top of them.
The system surfaces the disagreement, names both sources, and does not silently pick one. Better still, it tells the user who can resolve it.
A single clean answer that hides the conflict. Users will trust it, act on it, and discover the other version when it is expensive. The fix is often in your data governance, not in the AI system, and finding that out in a pilot is a good outcome.
Whether the system survives contact with real work.
Pick five to ten people and have them use the system for their actual first hour of work on a Monday, with no script and no vendor in the room. Ask them to bring the tasks they were going to do anyway: chase an invoice, answer a customer, prepare for a meeting, check an approval. Record where they gave up, where they double checked in another system, and where they got something useful and moved on.
People finish real tasks faster or with less effort, and they can tell you why they trust or distrust a given answer.
Users who quietly return to the old way after day two. That is not a training problem. It is a signal that the system does not fit how work actually happens, and usually that is clearer on a Monday than in any benchmark.
If people are hopping between four systems to check the answer the assistant gave them, you have not removed the switching tax that fragmented work already charges. You have added a fifth system to it.
Part 4 · Measurement
What to measure, and what to ignore
A pilot produces a lot of numbers. Most are easy to collect and say little about whether the system is safe to roll out.
| Measure | What it tells you | Rough target to agree up front |
|---|---|---|
| Answer correctness, by department | Whether quality is even or hides a weak area | Set per use case; higher for legal and finance than for general lookup |
| Source correctness | Whether answers come from the right documents | Close to correctness; a big gap means luck |
| Permission violations | Whether access controls hold | Zero. Not a percentage |
| Correct refusals | Whether the system admits what it cannot answer | Agreed threshold on the no-answer set |
| Consistency across repeat runs | Whether the answer is stable | Same question, same result across at least five runs |
| Time to correct answer for users | Whether it saves effort once checking is included | Compared with the current way of doing it |
| Escalations and edits | How often humans fix or override the output | Falling over the pilot, not rising |
And these are the numbers to treat with suspicion: the count of documents indexed, which says nothing about whether the right ones are used; average user satisfaction, which rewards fluent answers whether or not they are correct; raw response speed, which matters only after the answer is right; the total number of questions asked, which measures curiosity; and any accuracy figure that does not say how it was measured and on whose questions.
Part 5 · Pilot to production
What breaks between pilot and production
A pilot is a small, friendly, well-supervised environment. Production is none of those.
The problems below rarely appear in pilot results, and every one of them has ended a rollout somewhere.
- Connectors drift. In a pilot, someone checks that the SharePoint, ERP or CRM connection is working. In production, a token expires or a schema changes, the sync quietly stops, and the system keeps answering from yesterday's data. Ask how you will know when a source stops syncing, and who gets the alert.
- Permissions change daily. People move teams, leave, and get temporary access. A system that copies permissions at one point in time will be wrong within weeks. Test how fast a change in the source system reaches the AI system, and test it again with a real offboarding.
- The corpus grows and gets messier. A pilot usually runs on a curated slice. Production adds duplicates, drafts, scanned PDFs and the folder everybody calls old. Quality that held on 5,000 clean documents may not hold on 500,000 untidy ones.
- Workflows are longer than demos. Each added step compounds error. At 95% reliability per step, a 20-step workflow succeeds only about 36% of the time end to end. Even 99% per step leaves you near 82%.
- Models change underneath you. A provider updates a model version, and behaviour shifts. Without your question set to rerun, you will find out from a user.
- Nobody owns it. In the pilot, a project lead watched everything. After rollout, the system sits between IT, the data owners and the business. Name one owner for quality and one for access before go-live.
- Feedback goes nowhere. Users flag wrong answers, and nothing changes because there is no route from the flag to the document owner. Close that loop and the system improves. Leave it open and users stop flagging.
The practical answer is to treat the pilot test set as a permanent asset. Run it on every change to a model, connector, permission setting or prompt, and ship only when it passes. That turns a one-time evaluation into a standing safeguard, and it pairs naturally with an architecture that reads live data from the systems you already run rather than from a copy that can quietly age.
Part 6 · Due diligence
Questions to ask your vendor
By now you know how the system behaves on your data. These test how the vendor behaves, which matters just as much once the contract is signed.
| Ask | Red flag in the answer |
|---|---|
| Can we run our own question set, including questions you have not seen, and see the raw results? | Only a curated demo, or results shown as a summary score |
| How are user permissions enforced at retrieval time, and how quickly does a change in the source system take effect? | An assurance that permissions are respected, with no mechanism or timing |
| What happens when the sources conflict, or when no answer exists? | Always returns an answer; no refusal behaviour to demonstrate |
| How does the system handle outdated documents and versions? | No notion of freshness, or advice to delete old files |
| How do we see which sources produced an answer, and can we audit past answers? | No citations, or logs only the vendor can read |
| Where does our data go, how long is it kept, and is it used for training? | Region unnamed, retention described as a limited period |
| What changes when you update the model, and how will we be told? | Updates happen silently; no version pinning |
| What does a failed or stale connector look like to us, and who is alerted? | Failures visible only in vendor dashboards |
| Who owns quality on your side after go-live, and what is the support path for a wrong answer? | A generic ticket queue and no named owner |
If a vendor welcomes these questions, that is a good sign. If they redirect to a certification or a security badge, remember that safety is a property of how the system is built and run, not a label on the box. The data residency and retention answers deserve their own session; we set out what to listen for in where does your data actually go when you use enterprise AI.
Part 7 · The takeaway
A short checklist to take into your pilot
Testing is not the step that slows a rollout down. It is the step that decides whether the rollout survives its first quarter.
A pilot that finds problems has done its job, and the cheapest place to find them is before a thousand people depend on the answers.
Test it on your own data, with your own questions.
Eerly AI Studio reads live data from the systems you already run and works within each user's existing permissions, so the permissions test above is one we are glad to run with you. Bring your own question set and we will go through the raw results together.
Book a demo →FAQ
Frequently asked questions
What enterprise teams ask us most about testing AI before a rollout.
How do you test an enterprise AI system before rolling it out?
Build a private question set of 100 to 200 questions taken from real work, each with an expert-approved answer and a named source document. Score four things separately: right answer, right source, right permissions and right refusal. Then run five targeted tests for stale documents, permissions, no-answer cases, conflicting sources and real use on a normal working morning. Repeat every question several times to measure consistency, and agree the stop conditions before the first result arrives.
What does 95% accurate actually mean for an AI system?
Usually it means a reviewer looked at a sample of answers and judged them good. That hides at least four different ways of being wrong. A system might be 92% right on answers, 80% right on sources and 40% right on refusals, and a blended score would report 85%. Ask what was counted, who wrote the questions and whether source correctness and permissions were scored at all.
How many questions should an enterprise AI pilot test set contain?
Around 100 to 200 is enough to see patterns by department and question type. Below 50, a handful of lucky answers can swing the result. At least a fifth of the set should be questions where the correct behaviour is to decline, and a portion should be held back from the vendor so there is a score that cannot be tuned against.
Why do enterprise AI pilots succeed and then fail in production?
Pilots run on a curated slice of data with a project lead watching. Production adds connector drift when a token expires or a schema changes, permission changes as people move teams or leave, a messier and much larger corpus, longer workflows where per-step error compounds, silent model updates, and no named owner for quality. The practical defence is to treat the pilot question set as a permanent regression suite and rerun it on every change.
How do you test whether an AI system respects user permissions?
Use test accounts at three levels, a general employee, a department manager and a senior administrator, and ask each the same sensitive questions about salary bands, board papers, M&A files and contracts. Then try indirect routes, such as asking for a summary of everything about a project. Restricted material should be neither quoted, paraphrased nor confirmed to exist. Finish by revoking a test user's access in the source system and measuring how long the AI system takes to follow.
What metrics matter in an enterprise AI pilot?
Answer correctness by department, source correctness, permission violations, correct refusals, consistency across repeat runs, time to a correct answer once checking is included, and the escalation or edit rate over time. Treat documents indexed, average satisfaction, raw response speed, total questions asked and any unexplained accuracy figure as weak signals.
What should you ask an AI vendor before signing?
Ask whether you can run your own unseen question set and see raw results, how permissions are enforced at retrieval time and how fast a change propagates, what happens when sources conflict or no answer exists, how outdated versions are handled, how answers are cited and audited, where data is stored and whether it trains models, how model updates are communicated, what a failed connector looks like to you, and who owns quality after go-live. Vague answers are the red flag.
Sources & further reading
- Gartner. Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027 (June 2025). Cancellation drivers cited: escalating costs, unclear business value and inadequate risk controls. gartner.com
- MIT Media Lab, Project NANDA. The GenAI Divide: State of AI in Business 2025 (August 2025), as reported. Approximately 95% of enterprise generative AI pilots delivered no measurable P&L impact. healthcareitnews.com
- Yao, S., Shinn, N., Razavi, P. and Narasimhan, K. (Sierra). tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains (arXiv:2406.12045, June 2024). Introduces pass^k, the probability that an agent succeeds on all k independent trials of the same task. arxiv.org
- Sierra. Benchmarking AI agents (2024). Reports the drop from single-run scores to pass^k across repeated trials on the retail domain. sierra.ai
Neel engineers the foundational infrastructure powering Eerly's agents, building the isolation, access controls, and data pathways that determine whether an enterprise AI system is secure and robust enough to handle real company data.