Customer enquiries often arrive with missing details. Someone writes “Can you do it next week?” without naming the service, the location or the date they have in mind. An assistant can sound confident while answering a different question from the one the customer intended. Before a pilot goes live, give it a small set of deliberately ambiguous messages. Check when it can answer from reliable information, when it asks a follow-up and when it should pass the conversation to a person.
Choose one boundary the team understands
Start with one workflow, such as answering service enquiries from approved descriptions and opening hours. Ask the staff member who handles those enquiries to name the facts they must know before giving a useful answer. For a request to arrange a visit, those might include the type of work, service area and preferred date. For a question about hours, knowing the branch may be enough. Keep the pilot's authority narrow. Giving a customer general information and confirming a booking are different jobs.
Write down where the assistant can obtain each fact. A current service page might explain coverage, but it might say nothing about availability next Tuesday. A tool that checks available appointments can answer only if it has successfully returned the requested location and time. Decide what the assistant should say when that tool is unavailable. Keep the customer message and the evidence used for the answer separate so a reviewer can tell what was supplied and what was inferred.
Build examples around everyday omissions
Create a few fictional messages, each missing one essential detail. “How much is it?” tests whether the assistant notices that no service was specified. “Do you cover this area?” tests whether the assistant asks for a location that is specific enough for the business to check. “Is Friday available?” tests whether the assistant can distinguish a preferred day from a confirmed appointment. Where the business offers several services, use a message whose wording could reasonably refer to more than one of them.
Add a paired example for each omission. In the second message, supply the missing fact. This shows whether the assistant's follow-up was necessary. An assistant that asks the same question after the customer already provided the answer is adding friction. A good follow-up identifies the missing fact and lets the customer respond in a short message. Avoid requesting personal information that is irrelevant to the decision.
Read the question and answer as a conversation
Review the first response without looking at the team's expected result. Ask whether the reply acknowledges what the customer wants and gives them a clear next step. For “Can you do it next week?”, a useful response can ask which service the customer means and which day they prefer. It should avoid promising a slot based on the wording alone. Once the customer answers, the assistant should carry that detail forward instead of starting the interview again.
Test a correction in the next turn. The customer might change the location or say that “Friday” means the following week. Check that the response uses the corrected detail, and that an earlier assumption does not survive into the handover summary. A good test also includes a message that names two alternatives. The assistant should preserve both when the customer is asking for a comparison, and ask them to choose only when the next action requires a single selection.
When planning this pilot with a Blackstone AI agency team, bring a few examples from your workflow with fictional details. Agree which responses need evidence, which need a clarification and which must be reviewed by staff before anything is promised to a customer.
Check when a person needs to take over
Give the assistant a request that falls outside the team's policy, such as a request for a special price or for work in an area with no documented rule. Ask the reviewer what a staff member would need to continue the conversation. A short handover can include the service being discussed, what the customer already supplied and the point the assistant could not resolve. It should make the uncertainty visible so the next person does not mistake a draft answer for a decision.
Test the transfer through the actual staging interface. An assistant that says it will pass a request to the team should do so only when the workflow can confirm that the transfer happened. Check that staff can find the request and that the customer sees a truthful message about what to expect. If the transfer fails, provide a practical route to contact the business. Do not leave the customer waiting for a reply that has no destination.
Score the behaviour you can observe
Use a small table with the customer message, what is missing, the expected action and what actually happened. For each reply, record whether it answered with approved information, asked for the right detail, made an unsupported commitment or handed the case to staff. Keep examples where the wording was fluent but the action was wrong. These are useful in a review because they show how a reassuring tone can conceal an operational mistake.
Include ordinary questions that the assistant can answer directly. If every test is ambiguous, the safest-looking result may be a follow-up to every message. A well-scoped assistant should also know when it has enough information to respond. Ask a second reviewer to use the table independently. Discuss any disagreements and revise the expected actions before treating the cases as acceptance checks.
Keep the tests with the workflow
Save the examples and expected actions beside the notes that explain the approved sources and handover rules. Record the version of the workflow and the date of the check. Repeat the relevant cases after changing service information, switching the model or adding a tool that can take an action. If a rule changes, update the examples at the same time so they still reflect what the business can deliver.
Finish by having a staff member take one fictional enquiry from the first unclear message to a useful outcome. Watch the whole exchange, including the clarification and the final reply. The pilot is easier to judge when the team can see exactly how a customer moves from an incomplete question to a supported answer or a conversation with the right person.
Prepared by Blackstone Intelligence Sarawak for teams evaluating customer-enquiry assistants.