WhatsApp chatbot or AI agent: six tests to tell which one you have

Some chatbots are sold as AI agents, and some agents present themselves as just another chatbot. The name doesn't help. Testing does: six tests with messages anyone can type from their phone, without seeing the code or asking for a guided demo.

What is the difference between a chatbot and an AI agent?

A chatbot walks a path laid out in advance. An agent interprets what the person says, decides what to do and can look up or change data in other systems. A chatbot can use a language model to draft better replies and still be a chatbot, because the path is still fixed.

What tests should you run?

1. Change the subject midway. Start an inquiry, and in the third message ask about something else. An agent answers the new question and goes back to the first one if it makes sense. A rule-based chatbot returns to the menu or repeats the question.

2. Ask for two things in one message. "I want to change my address and find out when my order arrives." An agent separates the two. A chatbot usually answers one and loses the other.

3. Ask a question that isn't on any list. Something reasonable but unusual. An agent tries to resolve it or says clearly what it can't do. A chatbot replies with a generic sentence.

4. Ask about the status of something of yours. An order, a payment, an appointment. If it answers with real data, it's connected to a system. If it only tells you how to check, it isn't.

5. Ask to talk to a person. A good agent hands off without arguing and with a summary. Check whether the person receiving the conversation already knows what happened or whether you have to repeat everything.

6. Correct it. Tell it that it got something wrong and give it the right detail. An agent adjusts its answer. A chatbot repeats the previous one.

What does the result mean?

• Four or more tests passed: it's probably an agent with access to data.
• Two or three: it may be a chatbot with a language model on top. It works for general inquiries.
• One or none: it's a rule-based chatbot. It works well if the universe of inquiries is small and stable.

None of the three results is bad in itself. What matters is that what you buy matches what you need.

Why does it matter since October?

Because Meta charges for service messages, and a chatbot that goes around in circles sends more messages per inquiry than one that resolves. The number of outbound messages per resolved inquiry has become a cost figure, as well as a quality figure. You can write it down while you run the tests.

What happens inside each one?

A rule-based chatbot compares what the customer writes against a list of intents or keywords and returns the associated answer. It doesn't decide: it looks for the closest match. A chatbot with a language model drafts better, but the path is still the one it was given.

An agent works as a loop. It reads the message, decides what it needs to know, queries a tool (a system, a knowledge base, a calendar), looks at the result, decides the next step and only then answers or acts. If the tool returns an error, it can retry or hand off. That ability to decide based on what it finds is what makes the behavior look different in the tests above.

How to run the six tests, with concrete messages

To make them comparable across vendors, use the same messages with all of them. Here are a few examples as a starting point; replace them with inquiries from your own business:

Test 1, change of subject. First message: "I want to know the price of the basic plan." Second: "Oh, and before that, are you open on Saturdays?" Third: "Back to the price, does it include support?" A good result answers all three and remembers which plan was being discussed.

Test 2, two requests in one message. "I need to change the delivery address and find out when my order arrives." A good result separates the two requests and resolves them, or says which one it can't resolve and why.

Test 3, question off the list. Something reasonable that nobody anticipated: "If I pay in two installments, can I swap the product afterwards?" A good result tries to answer with what it knows or says clearly that it can't and hands off. A bad result replies with a generic sentence that has nothing to do with it.

Test 4, status of something of yours. "Where is my order?" with a real order number. A good result returns the status. A bad result explains how to check it.

Test 5, asking for a person. "I want to talk to someone." A good result hands off without arguing. Then check on the other side: does the person receiving the conversation see a summary or do they have to reread everything?

Test 6, correction. "No, the address is different: Av. Corrientes 1234." A good result updates it and confirms. A bad result repeats the previous address.

How do you record what happened?

A spreadsheet with three columns is enough: the message you sent, what it answered and whether it was correct. Add a fourth with how many outbound messages it sent to resolve each test, because that is the figure that later turns into cost. With fifteen minutes and two vendors you have a comparison worth more than any product page.

What does choosing the wrong tool cost?

Getting it wrong on the low side (buying a chatbot when you needed an agent) shows up as your team still handling almost everything, because handoffs are high and the inquiries that need data don't get resolved. Getting it wrong on the high side (buying an agent for general information inquiries) shows up in the cost: you pay for capacity you don't use. Both can be corrected, but the first is discovered late: it usually appears after several months of contract.

What questions should you ask the vendor after the tests?

• Which systems can it look up and change today, and which ones need development?
• How many outbound messages does it send on average per resolved conversation?
• What happens when it doesn't know? What does the handoff look like on the other side?
• How are conversations reviewed and who reviews them?
• What counts as a resolved conversation for billing?

If the vendor answers these five with real examples, you already know more than from any demo.

What should you do depending on how many tests it passed?

With four or more passed, the next step is to measure costs: how many outbound messages it sends per resolved conversation and what the billing includes. With two or three, define which inquiries stay unresolved and how much each one costs in team time, because that is what you would be paying to move to an agent. With one or none, ask yourself whether the universe of inquiries is small and stable: if it is, a rule-based chatbot is still a valid and cheap option. And if it isn't, you already know you need something else.

What changes when voice is added?

The six tests apply just the same to a voice agent, with one complication: answers happen in real time and transcription errors are more frequent than typing errors. Dictating an email, an order number or a surname is the hardest case. A well-built voice agent confirms what it understood ("Is that a, b, c at…?") or opens a written channel in parallel so the customer can type the detail. To evaluate one, add a seventh test: dictate a tricky detail and watch how it handles it.

What does this look like in a real operation?

A company that receives customer inquiries on WhatsApp and leads from ads usually has both needs: a chatbot or agent that answers frequent questions and one that qualifies leads and checks orders. Some companies start with a rule-based chatbot for the most frequent questions and, as they discover how many inquiries stay unresolved, move to an agent for what requires data and actions. It's a reasonable path. What is worth avoiding is buying an agent without first measuring which inquiries arrive, because then there is no way to know whether it was worth it.

What happens when the customer sends a voice note or an image?

It's an everyday occurrence on WhatsApp and a good extra test. A rule-based chatbot usually ignores the voice note or asks for the message to be typed. A more complete agent transcribes the audio, interprets the image when it's relevant (a photo of a damaged product, a receipt) and responds accordingly. If your operation receives a lot of voice notes, add this test before deciding: send a ten-second voice note with a clear inquiry and watch what it does. With per-message billing, a misread voice note costs twice: the wrong answer and the correction.

How do you explain these tests to whoever decides the purchase?

With one sentence: "We're not looking at what the vendor says it does, we're looking at what it does with your messages." The six tests take less than an hour, require no technical knowledge and produce a spreadsheet that can be shared with anyone on the team. The person deciding the purchase doesn't need to understand how a language model works: they need to see, with their own inquiries, the difference between what it resolves and what it doesn't. And if the vendor refuses to let you run the tests with your own messages, that is an answer too.

What should you do with what you found?

If you're looking for an agent and what you have is a chatbot, you don't need to change everything. Write down which inquiries stay unresolved and how much they cost in team time. That is the argument for deciding whether it's worth taking the step, and which process to start with.

Tool for this post

The six tests, with a score

Try it on your WhatsApp or a vendor's and check what happened.

I changed the subject midway and it handled both things well (+1)I asked for two things in a single message and it resolved both (+1)I asked an off-list question and it handled it sensibly (+1)I asked about the real status of something (order, payment, appointment) and it answered with data (+1)I asked for a person and it handed off with a summary (+1)I corrected it and it adjusted its answer (+1)

0 of 6

Rule-based chatbot

It works if the universe of inquiries is small and stable. For varied inquiries or ones that need customer data, it will fall short.

This object also has a page of its own, in case you want to send just that.