How to choose an AI agent for customer service

Two years ago, adding "AI" to a support tool meant a decision-tree chatbot with a friendlier name. Today almost every help desk and live chat vendor offers an AI agent that reads your content and answers in natural language. On a demo, they all look impressive.

The differences show up later: when a customer asks something slightly unusual, when the agent needs order data, when a frustrated customer wants a person, or when the first invoice arrives in December. This guide is about finding those differences before you sign.

What an AI agent should actually do

A useful customer service AI agent does five things well:

1. Understands the question, including typos, slang, several questions in one message and other languages.
2. Answers from your content, not from the general internet, and says so when the content doesn't cover it.
3. Takes simple actions, such as checking an order, booking a slot or collecting return details.
4. Hands over cleanly to a person, with the full context, when it should.
5. Shows you what happened, so you can see what it resolved, where it failed and which content is missing.

If a tool is weak in any of these, the others don't matter much.

The six differences that matter

1. Knowledge sources

Which content can the agent learn from? Look for: your website, help center, uploaded documents, Q&A pairs you write yourself, and product catalogue data for stores. Check how quickly it picks up changes. If you update the returns policy at 9am, the agent should know by lunchtime.

2. Actions and integrations

Answering "what is your returns policy?" is easy. Answering "where is my order #4821?" requires a connection to your store or backend. Ask what actions are available out of the box (order status, cancellations, address changes, bookings) and how custom actions are built: visually, or only with developers and APIs.

3. Handoff to humans

Ask the vendor to show you a handoff on a live demo. You want:

• automatic handoff when the agent is unsure or the customer is upset;
• the customer can ask for a person at any time;
• the human sees the whole conversation and customer details;
• sensible behaviour outside business hours (collect details, set expectations, create a ticket).

4. Guardrails and tone

Can you tell the agent which topics to avoid, how formal to be, and when never to answer (legal threats, medical questions, refunds above a limit)? Can you preview its answers before going live? Does it say clearly that it is an AI?

5. Channels and languages

If your customers write on Instagram and WhatsApp, an AI agent that only works in the website widget solves half the problem. Check every channel you use, and test in each language you serve, not just English.

6. Reporting

At a minimum you want resolution rate, handoff rate, satisfaction on AI conversations and a list of questions it couldn't answer. That last list is your content to-do list.

Understanding AI agent pricing

Pricing is where AI agents differ most, and where surprises happen.

Pricing model	How it works	Good for	Watch out for
Per conversation	You pay for each conversation the AI takes part in	Predictable budgeting	You also pay when the AI fails and hands over
Per resolution	You pay only when the AI closes the issue	Paying for results	Hard to forecast; definitions of "resolved" differ
Per seat with AI included	AI bundled into agent licences	Small teams with low volume	Limits or fair-use caps
Add-on on top of plan	Base plan plus AI add-on, sometimes plus usage	Existing customers of a suite	Total cost can be several times the headline price

Do the maths with your own numbers. Take your monthly conversation volume, multiply by your expected automated resolution rate (be conservative at first), and price each vendor at that volume. Then do it again for your busiest month. A store that triples volume in November should know what the AI will cost that month before it happens.

How to run a fair two-week test

Never pick on demos alone. Shortlist two or three tools and test them the same way:

1. Collect 50 to 100 real questions from your inbox, covering your top topics plus a few awkward ones (complaints, edge cases, questions in other languages).
2. Give each tool the same content: your help center URL and the same Q&A pairs.
3. Ask every question and score each answer: correct, partly correct, wrong, or correctly handed over.
4. Test the handoff and an action such as an order lookup.
5. Go live on a small share of traffic for a week, for example website chat during business hours, and read every transcript.
6. Compare resolution rate, wrong answers, customer ratings and projected cost.

A "correctly handed over" is a good result. A confident wrong answer is the worst one.

Red flags during evaluation

• The vendor can't show you which content an answer came from.
• There is no simple way for customers to reach a person.
• "Resolution" is counted when the customer simply stops replying.
• AI is only available on the most expensive plan, plus an add-on, plus usage.
• Setup requires weeks of professional services for a basic use case.

Which one should you choose?

• Small or mid-size business, especially an online store: start with Tidio. Lyro, live chat, Flows and the inbox work together from day one, and you can test it on the free plan.
• Established brand with high volume and no time to tune the AI: look at Tidio's Premium plan. Tidio's team runs Lyro as a managed service, with a guaranteed 50% resolution rate and billing per resolution.
• SaaS company with deep documentation: compare Intercom Fin with Tidio on your real questions.
• Enterprise already on Zendesk: evaluate Zendesk's own AI first, and Fin as an alternative that works on top of it.
• Shopify brand already on Gorgias: try its AI agent before adding another tool.

Whichever you choose, the content you feed it matters more than the logo. For the rollout itself, follow our guide on how to automate customer support.