An LLM support chatbot: when it pays off and how to launch it

An LLM support chatbot pays off when many requests repeat — order status, delivery, returns, common product questions — and the answers exist in your knowledge base or systems. It works when it answers from your documents, takes simple actions through APIs and hands off to a person in time; otherwise it is polite but useless.

Here is how to tell whether a bot will pay off, how it works and how to launch it without putting customers at risk.

When does a support chatbot pay off?

A chatbot pays off when there are many requests, a large share of them are routine, and the answers can be found in documents or retrieved from company systems. If most requests are unique and need a human decision, the bot will save little and may irritate customers.

How to assess the potential:

1. Export several months of support requests and tag them by topic.
2. Count the share of routine questions that have a ready answer or data in your systems.
3. Estimate how much agent time those questions take now.
4. Compare that with the cost of building, running and maintaining the bot.

We covered how to estimate model call costs in AI feature cost.

How does an LLM support chatbot work?

A modern support bot consists of a model, knowledge base retrieval, a set of tools for acting in company systems and rules for handing off to a person. The model phrases answers and decides which tool to call, but relies on company data rather than its general knowledge.

Component	What it does	Example
Knowledge base retrieval	Finds articles and rules relevant to the question	Return policy, delivery times
Tools	Fetch data and take actions via APIs	Order status, changing the delivery address
Model	Understands the question, picks a source, writes the answer	An answer citing the relevant rule
Human handoff	Passes the conversation to an agent with a summary	Complaints, complex cases, customer request
Logs and ratings	Stores conversations and customer ratings	Material for improvements

Knowledge base retrieval follows the RAG approach — covered in detail in RAG: build an AI assistant on your documents.

Photo: Unsplash

Tools: a bot that does more than talk

For the bot to answer "where is my order", it needs data access through a limited set of tools. Tools are described by a schema, and permissions are checked on your server, not by the model:

{
  "name": "get_order_status",
  "description": "Status and expected delivery date of the current customer's order",
  "input_schema": {
    "type": "object",
    "properties": {
      "order_number": { "type": "string", "description": "Order number, for example 10423" }
    },
    "required": ["order_number"]
  }
}

The server runs the tool only for the customer's own orders: even if the model asks for someone else's order, the permission check blocks the request.

How do you launch a bot without putting customers at risk?

Launch the bot in stages: first as suggestions for agents, then on a share of requests or one channel, and only then for all customers. Each stage produces real quality data before the bot answers everyone.

Launch stages:

1. Agent suggestions. The bot drafts an answer; an agent reviews and sends it. Quality data accumulates with no risk.
2. A pilot on part of the traffic. The bot answers on its own for a limited share of requests or in one channel.
3. Full launch with fast human handoff and quality monitoring.
4. Continuous improvement by reviewing conversations where the bot erred or handed off.

Rules that protect customers

• the bot says honestly that it is a bot and always offers a way to reach a person;
• if the answer is not in the knowledge base, the bot does not make one up — it hands off;
• complaints, refunds and disputes go straight to a person;
• personal data is handled by the rules and never ends up where it should not.

How do you measure the bot's quality?

Measure the bot's quality by the share of requests resolved without an agent, customer satisfaction and the share of wrong answers found in spot checks. A high "resolution" rate is worthless if customers leave unhappy.

What to track:

• the share of conversations closed without a handoff;
• customer ratings after bot and agent conversations;
• the share of wrong answers in a weekly sample;
• time to resolution;
• handoff reasons — your list of what to teach the bot next.

We covered the general architecture of AI features — gateway, limits, validation — in how to integrate an LLM into your product.

In short

• A bot pays off with a large flow of routine requests whose answers exist in documents and systems.
• It answers from the knowledge base, takes simple actions through tools and hands off to a person in time.
• The server checks permissions, not the model: the bot sees only the current customer's data.
• Launch in stages: agent suggestions, a pilot, then full launch.
• Measure resolution rate together with satisfaction and answer accuracy.