Questions to Ask an AI Customer Service Vendor Before You Commit to a Pilot
November 18, 2026
Last Updated: 2026-11-17
The questions to ask an AI customer service vendor before committing to a pilot fall into seven areas: scope, customer data, security, human handoff, testing, metrics and stop criteria. An answer only counts if your team can check it before a single customer talks to the agent. That means checking audit reports for their date, period and scope, testing the agent on your own past conversations, getting the resolution rule in writing and watching a live handoff. Sign the stop and rollback plan before day one, and calculate the pilot cost per resolved conversation.
It's the week after the second demo. The vendor's agent answered every question the sales engineer typed, passed a mock refund request to a "live agent" without a hitch, and the last slide proposed a pilot starting next month. Your security lead has already sent a list of questions. Legal wants to know who's responsible if the agent gets a refund policy wrong. Your team leads want to know whether customers will end up stuck in a loop. And you're the one who signs off.
The demo showed the agent answering the questions the vendor picked. The pilot has to show it answering yours, so you need to set that up before you commit.
This guide is for heads of support, CX and contact center operations at mid-size and large US companies who are about to sign off on an AI agent pilot, with security, legal and procurement reviewing it alongside them. You'll leave with the questions to ask, what a verifiable answer looks like for each one, and a stop plan you can sign before the agent sees a real customer.
What should you ask an AI customer service vendor before committing to a pilot?
Ask the questions that turn a demo into evidence, grouped into seven areas (scope, customer data, security, human handoff, testing, metrics and stop criteria).
A useful AI customer service vendor evaluation adds three things to every question before anyone signs: what evidence counts as an answer, how your team will check it, and what decision that answer lets you make.
Plenty of published lists of questions to ask AI vendors already exist, and some of them ask for proof. The table below puts the evidence, the check and the decision next to every question. Its last row ties every answer to a stop plan that you and the vendor sign before the pilot starts.
A vendor answer only counts once your team can check it and tie it to a decision to go, change the scope or stop.
| Area and question | Evidence that counts | How you check it | Decision it unlocks |
|---|---|---|---|
| Scope. Which questions, channels, hours and actions will the agent handle? | A written scope built from your own contact logs, with excluded topics named | Compare the scope with a sample of recent conversations from your queue | Sign off on the scope, or narrow it before day one |
| Customer data. What will the agent read, and where is it processed? | A data flow listing knowledge sources, connected systems, any third-party model, retention and deletion | Have the vendor show each source, its last sync date and every system the agent can reach | Approve each source and system, or take it out of the pilot |
| Security. Which independent audits cover the service you're buying? | Audit reports with a report date, an audit period and a stated scope | Match the scope line in each report to the product in the pilot | Submit it for security review, or request the missing report |
| Human handoff. What happens when the agent can't answer? | A live handoff on one of your own tickets, showing what the person receives | Time the wait, read what lands on the human agent's screen and note whether the customer has to repeat information | Accept the handoff design, or add it to the stop criteria |
| Testing. How will the agent be tested before customers see it? | A test run on your own past questions, including the same question asked several ways | Score every answer with your own reviewers against your own sources | Move to a limited rollout, or extend testing |
| Metrics. How does the vendor count a resolution? | The resolution rule in writing, with its denominator | Recount a sample of "resolved" conversations by hand | Agree on the success criteria and how the total cost per resolved conversation is calculated |
| Stop criteria. Who can switch the agent off, and what happens then? | A signed stop and rollback plan with named owners and a manual fallback | Rehearse one pause before launch, with the vendor on the call | Start the pilot, or hold it until the plan exists |
The last row connects the others. Every "add it to the stop criteria" in the decision column ends up in that plan, so by the time the pilot starts, you already know which answer would end it. The NIST AI Risk Management Framework is voluntary and sets no pass mark for any of this. It gives your security and legal reviewers a shared language, from third-party risk to who is responsible for deactivating a system.
What will the AI customer service pilot cover and exclude?
Start with your own logs. Pull the routine questions your team answers every day. Check which channels they arrive on and when they peak. Then write the pilot scope around those questions.
The document should specify the share of conversations the agent will see, the systems it may only read, and the actions it may take (an order lookup, a booking change). It should also identify the topics that stay with people from day one and who on your team is responsible for each of those decisions.
AI pilot programs built on the vendor's showcase use cases can only prove the showcase.
A pilot scope is ready when every field comes from your own logs and both sides have signed it.
Ask how the vendor limits exposure while the pilot runs. HubSpot's documentation describes deploying its customer agent to a single channel or to a small percentage of conversations, and setting the days and hours it responds. Sierra describes rolling a release out to a portion of customer traffic and rolling back just as fast if something unexpected appears. Make sure the scope document names the mechanism and the person allowed to change the percentage.
Then ask where you can try the agent before a customer can reach it. indigitall documents a client-specific space for testing chatbot responses before channel integration, with two-factor sign-in, and clients in the US and Latin America work on its AM1 platform. Confirm the testing space's isolation and access controls for your deployment. Its AI Voice page describes four stages (design, test in a sandbox, scale and optimize), with test calls run in the sandbox on your own scenarios. If you're still figuring out which conversations a bot should handle, indigitall's guide to chatbots for customer support is a broader place to start.
What customer data will the AI agent read, and where will it go?
Request the data flow on one page. It should show the knowledge sources the agent answers from and how often each one syncs. It should also identify the systems the agent can query in real time and whether any third-party language model sees your customers' messages. Finally, it should explain how long data is kept and how it gets deleted.
Security and legal will want this page before anything else. In banking, healthcare or any other regulated sector, it's the page that determines whether the pilot goes ahead at all.
Every arrow on this diagram is a question for the vendor, and every box needs a written answer before the pilot.
NIST's generative AI profile suggests contracts and service level agreements that spell out content ownership and usage rights, and it folds vendor due diligence into the way an organization acquires a generative AI system. It's a voluntary framework and doesn't certify anyone. Use it as a checklist for the contract review and as vocabulary both sides already share.
indigitall's AI agents are a concrete example of a checkable answer. They use retrieval-augmented generation trained only on the content the client provides, with nothing from the public internet. The AI agent documentation also says the client decides what information the agent uses.
Sources can be PDF or TXT files, web addresses or sitemaps, all in the same language. The sources settings show how often each one re-syncs and when it last did. The agent reaches systems such as a CRM, ERP, inventory, payments or a healthcare record through the Model Context Protocol (MCP).
On the same AI agents page, indigitall states that customer queries are not sent to OpenAI, Anthropic or Google for processing by a third-party language model. It also states that the reasoning layer runs on its own infrastructure with your data. Its security documentation, covered in the next section, lists Google Cloud as the infrastructure the platform is built on.
Confirm the full data flow and infrastructure for your deployment.
Each connected system also requires integration work. Ask which systems the pilot really needs, who builds each connection, and how much time and support the vendor expects each connection to require, including support from your team.
Retention and deletion belong on that same page. Ask how long conversation logs and data pulled from connected systems are stored, where they're stored, and how you'd get them deleted when the pilot ends. Get those answers in writing, whatever the vendor's certifications say.
How can you verify the vendor's security before the pilot?
Get the reports, with their dates. A security page full of badges tells you which audits a vendor has passed at some point. A dated report tells you when the audit happened, which period it covered and which service it applied to, and that's what your security team needs before it signs off.
The scope is the detail that's easiest to overlook. An audit can cover one product and leave out another, so match the scope in each report to the exact service in your pilot. Check access and encryption next, meaning role-based access with multi-factor authentication for the people who configure the agent, and encryption in transit and at rest. A healthcare buyer should also ask whether the vendor signs Business Associate Agreements and which service its HIPAA audit actually covers.
indigitall's security documentation shows what a checkable answer looks like.
It lists a SOC 2 Type II report dated January 20, 2026, covering an audit period from August 15 to November 14, 2025. It also lists ISO/IEC 27001:2022 certification valid until July 23, 2029, and a GDPR verification valid until April 17, 2027.
Its HIPAA audit has a final report dated May 22, 2025, and covers the encrypted push notification service. The page also says indigitall signs BAAs with US healthcare organizations.
The same page lists role-based access control with multi-factor authentication, TLS 1.2+ in transit, AES-256 at rest and infrastructure built on Google Cloud.
Claims about accuracy deserve the same treatment as claims about security. When the FTC announced Operation AI Comply on September 25, 2024, it said there is "no AI exemption from the laws on the books," and its complaint against DoNotPay alleged the company hadn't tested whether its chatbot's output matched the level of a human lawyer. The agency's final order against Workado, announced on August 28, 2025, requires "competent and reliable evidence to support the claim at the time it is made" for claims about its product's effectiveness. If a vendor quotes a resolution or accuracy rate, ask for the method and the evidence before the pilot, and treat a number that comes with neither as marketing.
What happens when the AI customer service agent can't answer?
Have the vendor demonstrate it on one of your own tickets. A good handoff test answers four questions in a single run:
- How does the agent know it can't answer?
- How long does the customer wait for a person?
- What does that person see when the conversation lands (the history, the customer profile, the steps already tried)?
- Was the customer told from the start that they were talking to an AI?
A handoff that makes the customer start over only delays the resolution it was meant to deliver.
Test whether the person receives the history, relevant profile and steps already tried, and whether the customer needs to repeat information.
Regulators and lawmakers are already treating the handoff as a compliance question. The CFPB's report on chatbots in consumer finance, published on June 6, 2023, described "doom loops" that are often caused when an issue falls outside a chatbot's limited capabilities, warned that offramps to human support can be blocked by unreasonable waits, and said inaccurate information about a financial product "could be catastrophic." It added that chatbots must comply with all applicable federal consumer financial laws. California's AB 1609, signed on September 28, 2026, requires large businesses to make a good-faith effort to connect a customer to a human within 15 minutes of a request, or schedule an appointment, and to disclose when the representative is a chatbot, according to the governor's announcement.
indigitall's AI agents page says its agent hands off to a live agent in the Contact Center with the full conversation history, the customer profile and the attempted resolution steps already loaded, which the company sums up as "No cold handoffs." Check that claim on your own tickets, as you would any other. The agent can transfer to an unlimited, configurable number of human agents. The person taking over a voice call gets a real-time transcript and a summary of the context, and the voice agent tells the customer at the start of each call that they're speaking with an AI agent.
Then ask how you'll monitor handoffs while the pilot runs. You want queue length, average and maximum wait, and the ability to read any conversation and transfer it. indigitall's supervisor view shows those statistics in real time, and its agent panel lets an agent route an ongoing chat to another agent or department.
How should you test the AI agent before real customers see it?
Test with your own past questions in an environment customers can't access. Take a set of real conversations from your logs, keep the awkward ones, ask the same question several ways, and have your own reviewers score the answers against your sources. NIST's generative AI profile frames this as evaluating "claims of model capabilities using empirically validated methods," and it names confabulation, a model confidently presenting false content, among the risks it covers. Vendors talk a lot about hallucination prevention. A test run on your own data is the version of that claim your security and legal teams can actually check.
Vendor documentation shows what a serious test looks like. Salesforce recommends using its Agentforce Testing Center only in a sandbox environment.
Intercom describes batch tests of up to 50 questions for eligible existing workspaces. Since November 5, 2026, new workspaces use Evals instead. Evals simulate multi-turn conversations and can be rerun as a regression suite after a change. Teams can recreate a real conversation to test it again.
HubSpot lets admins generate batch test questions from existing customer conversations before the agent goes to live channels. Zendesk advises asking realistic questions, asking the same question in different ways, and using two kinds of testers: one focused on process and one on experience. Sierra describes voice simulations with accents, background noise and interruptions, built to catch the problems that only show up in calls.
Turn that into commitments and write them into the pilot plan.
Ask whether you can load a batch of your own questions or only type them one by one. Find out who scores the answers and against which rubric. Also ask whether the tests run again after every change to the knowledge sources and who on your team gets access to the test environment.
Guardrails belong in the same test run. indigitall's generative AI chatbot documentation lists guardrails you can check in the testing space before channel integration. The bot never returns HTML code, it limits the length of individual words in its answers, and its welcome message identifies it as a bot. According to its testing documentation, the chatbot restricts access until the following day when a user exceeds the default limit of 10 inappropriate questions in one day. The limit is configurable through the chat channel's API.
Which metrics decide whether the pilot works?
Write five metrics into the pilot plan, each with a baseline you measure yourself before day one:
- The resolution rule and its denominator
- Fallbacks
- Recontact on the same issue
- Wait time after a handoff
- Cost per resolved conversation
Agree on the resolution rule before you agree on any target. Those five numbers, compared with your own baseline, become the pilot's success criteria in the contract annex.
The pilot works when these five numbers move against a baseline you measured yourself, under a resolution rule both sides signed.
Each vendor counts a resolution its own way, so get the definition in writing and decide which conversations count as resolved in your pilot. Zendesk separates a verified resolution from a contained one. For customer-agent replies, HubSpot counts a resolution when the agent provides a sourced answer or performs an action and there's no qualifying handoff within 72 hours of the visitor's last response. Lead qualification is a separate resolution path. Salesforce reports a deflection rate, the share of sessions that ended without escalation. That's why there's no meaningful average resolution rate for AI customer service agents across industries. The published numbers don't count the same thing.
Quality and customer experience need their own signals. Track fallbacks (the questions the agent couldn't answer) and whether they drop week by week as you fix the knowledge sources. Track recontact too. A customer returning about the same issue can signal an unresolved problem. Review those conversations before you classify them as unresolved. Calculate the total pilot cost per resolved conversation using an agreed resolution rule, and review a sample to check whether customers actually received help.
indigitall's chatbot detects when a question goes unanswered, reports the most frequently asked topics and the fallbacks, and lets customers rate the conversation with emojis. Clients can also set a monthly spending limit and receive email alerts at 80% and 100% of it, which keeps the cost of a pilot visible while you measure it. Ask for access to the underlying conversations, whichever tool you pilot, so your team can recount a sample of "resolved" ones by hand.
When should you pause or stop the AI customer service pilot?
Decide that before day one, and sign it.
A stop and rollback plan should name the signals that pause the agent and who can switch it off. It should explain what happens to conversations in flight and how customers get back to a person while you fix the problem. It should also name who signs the plan (support, security and the vendor).
NIST's generative AI profile asks for protocols that make sure a system is "able to be deactivated when necessary," along with specific deactivation criteria that are reviewed regularly. The AI RMF adds mechanisms and assigned responsibilities to "supersede, disengage, or deactivate" a system that performs in ways inconsistent with its intended use.
The framework is voluntary and sets no thresholds. Your team picks those, and NIST gives the plan a structure your reviewers will recognize.
The pilot is ready to start when the plan that would end it is already signed by support, security and the vendor.
The thresholds are yours to set, but the signals are easy to name in advance. These are examples to adapt.
| Stop signal | What it looks like | Who decides | What happens next |
|---|---|---|---|
| Inaccurate answer on a sensitive topic | The agent gives a wrong refund, billing or eligibility answer | Support lead with compliance | Pause that topic, route it to people, fix the source |
| A person can't be reached | Customers who ask for a human wait longer than your plan allows | Contact center manager | Pause the agent on that channel until staffing recovers |
| Data outside the agreed scope | The agent reads or reveals data from a system that isn't in the scope | Security | Stop the pilot and review access |
| Fallbacks that don't drop | Unanswered questions stay flat after knowledge fixes | Support lead and vendor | Freeze any expansion and review the scope |
| A vendor claim you can't reproduce | A quoted rate doesn't hold on your own sample | Procurement | Hold the rollout decision |
Rollback needs a destination. NIST's profile treats fallback as something to test and manage, and notes that the fallback can be manual processing, so write down which queue takes over the agent's conversations, how quickly, and who tells the team. Rehearse one pause before launch with the vendor on the call, and measure how long it takes for every channel to return to human handling.
Plan the exit while you're at it. Ask what happens to your knowledge sources, test sets and conversation logs if you don't continue, and how they're returned or deleted. A pilot you can exit cleanly is a pilot your reviewers can approve faster.
If you're putting these questions to vendors right now, put them to indigitall too. You can ask for a demo and check every answer against its public documentation on security, knowledge sources and the handoff to the Contact Center.
FAQs: AI customer service vendor pilot
How do you test an AI customer service agent for hallucinations before a pilot?
Run the agent on a set of your own past conversations before any customer can reach it, ask the same question several ways, include questions it shouldn't answer, and have your own reviewers score each answer against your sources. NIST's generative AI profile calls the risk confabulation and recommends evaluating claims of model capabilities with empirically validated methods. Ask the vendor which content the agent answers from and whether you can restrict it to your own sources, then make an inaccurate answer on a sensitive topic one of the pilot's stop signals.
Can you configure an AI agent's guardrails yourself, or does the vendor set them?
It depends on the vendor, so ask which guardrails are fixed, which ones your team can change, and who changes them during the pilot. indigitall's generative AI chatbot, for example, never returns HTML code, limits the length of individual words, identifies itself as a bot in its welcome message, and restricts access until the following day when a user exceeds the default limit of 10 inappropriate questions in one day. The limit is configurable through the chat channel's API. Write every guardrail you rely on into the pilot scope and test each one before launch.
What observability should an AI customer service vendor give you during a pilot?
You need real-time visibility into conversations, handoffs and queues, plus regular reports on the questions the agent couldn't answer. A workable setup shows queue length and average and maximum wait, lets a supervisor read any conversation and transfer it, and reports fallbacks and the most frequently asked topics. indigitall's supervisor view, for example, shows agent and conversation statistics in real time. Before the pilot starts, agree on who reviews these reports, how often, and which numbers trigger a stop.
How do you know the AI agent hands off to a person at the right moment?
Define the moments in writing (a topic on your sensitive list, a customer asking for a person, a repeated unanswered question) and test each one on your own tickets before launch. Once the pilot is live, read a sample of conversations where the agent kept going and check whether it should have handed off. California's AB 1609, signed on September 28, 2026, asks large businesses to make a good-faith effort to connect a customer with a person within 15 minutes of a request or schedule an appointment, and the CFPB's 2023 report on chatbots in consumer finance warned about offramps blocked by unreasonable waits.
What data residency and certifications should you ask an AI customer service vendor about?
Ask where your data is stored and processed, which language models see your customers' messages, how long logs are kept and how they're deleted, and get every answer in writing. Ask for each certification report with its date, audit period and scope, and check that the scope covers the product in the pilot. indigitall, for example, lists a SOC 2 Type II report dated January 20, 2026, ISO/IEC 27001:2022 certification valid until July 23, 2029, and a HIPAA audit whose scope is its encrypted push notification service.
Should you start with a sandbox or a pilot before signing a long contract?
Start with a sandbox, then move to a limited pilot. Testing in an environment customers can't reach lets you check answers, guardrails and handoffs on your own questions without exposing a single customer. A pilot then puts the agent in front of a share of real conversations under a signed scope, written success criteria and a stop plan. Ask each vendor where testing happens and what it takes to move from the sandbox to the pilot.
Why do AI customer service pilots fail to reach production?
AI customer service pilots stall when the success criteria, the resolution rule or the stop plan were never written down, so the results can't support a decision either way. Other common causes are a scope copied from the vendor's showcase use cases, and integration work with your own systems that nobody planned for. Writing down the success criteria, the resolution rule and the stop plan before the pilot starts gives you a result you can take to a committee.


