About the client: an education company supporting exam candidates
Our client operates in the education sector and supports people preparing for a certification exam. Candidates need information at different stages, including registration, exam procedures, receiving results and appealing a grade. The company provided this information in PDF documents totalling several hundred pages, mainly in English. Finding the right answer therefore required customers to search through extensive materials themselves. The chatbot was intended to help them handle exam-related matters and reduce the support team’s workload.
The challenge: improve answers before releasing the chatbot to customers
Candidates needed to be able to describe their situation in their own words and receive clear guidance within around 4–5 seconds. Answers also needed to be complete. If a procedure applied only under certain conditions, the chatbot needed to explain them.
TTMS was responsible for improving information retrieval in the existing solution, which had been built using CrewAI, a framework for developing AI agent-based systems. The chatbot used RAG (Retrieval-Augmented Generation) to generate answers from retrieved passages in the client’s documentation.
Our analysis showed that the text chunks supplied to the model were too large, making them harder to match to a question. Text extraction from multi-column PDFs caused another problem: content from adjacent columns was being mixed together. Improving the chatbot therefore required examining what information the model received and whether it preserved the meaning of the source document.
Solution and implementation: answers with the necessary context
We changed how the documentation was divided into chunks, preserving the connections between sentences, paragraphs and pages. When the system found a sentence relevant to a question, it could also retrieve the surrounding paragraph or the entire page. This gave the model the information needed to explain a rule along with the conditions under which it applied.
This approach allowed us to use the client’s existing materials and retain classic RAG. According to Jakub, the additional storage and processing costs were acceptable for this volume of documentation.
We evaluated the changes using an internal set of questions covering the full scope of the materials. We also included real user questions collected anonymously during an earlier prototype launch. This helped us assess answers to questions phrased in the way customers actually asked them.
We explain the technical approach in our article “RAG for chatbots using CrewAI – notes from a TTMS Tech Talk”.
Results: around 98% answer accuracy in an internal test
According to the data shared by Jakub Kraśniewski, answer accuracy increased from around 70% to around 98%, an improvement of approximately 28 percentage points. The solution also maintained a fast response time. This result applies to the internal test set prepared for the project. It confirmed the improvement in answer quality and helped prepare the tool for the next stage: release to a subset of customers. The actual reduction in the support team’s workload had not yet been assessed.
What comes next? Evaluating the chatbot in everyday candidate support
After completing internal testing, the company planned to make the chatbot available to a subset of customers. This stage will help establish whether candidates receive enough assistance to resolve their enquiries and need to contact a support adviser less often. The assessment will draw on chatbot usage data and support enquiries.
Let’s talk about a chatbot for your company
Would you like to make company documentation easier for customers or employees to use? Or do you already have a chatbot whose answers need improvement? Talk to the TTMS team. We will review your materials and users’ needs to recommend a solution that fits your business.
Where should you start with a customer service chatbot project?
Start by reviewing support enquiries to identify the questions customers ask repeatedly. Issues for which the company has clear, documented answers are good candidates for the initial scope. Next, define who the chatbot should help and what users should be able to accomplish through a conversation. This makes it easier to set the scope of a pilot and decide how to assess its results. Involve customer service staff early, as they understand common misunderstandings and situations that require an individual decision. A scope prepared this way gives the implementation team a concrete starting point.
Can you improve an existing chatbot without rebuilding it from scratch?
In many cases, individual components of an existing solution can be improved, although the extent of the changes requires an initial assessment. The team should review examples of incorrect answers and identify where the problem occurs. The cause may lie in document preparation, information retrieval or how the model uses the retrieved content. The limitations of the current platform and access to its configuration also matter. Before deciding to rebuild, it is worth comparing the effects of selected changes using the same set of questions. This helps establish whether further development of the existing tool is justified.
What determines the cost of implementing a chatbot that uses company documents?
The cost depends partly on the condition of the source materials, the integrations required and the tool’s operating requirements. Documents with complex layouts or inconsistent content may need more preparation. Connecting the chatbot to company systems is a separate task, particularly when access to information depends on user permissions. The budget should also cover testing, support after launch and knowledge base updates. Ongoing costs depend on conversation volumes, the models selected and how questions are processed. An estimate should therefore distinguish between the cost of building the solution and the expected cost of maintaining it.
How do you plan a customer chatbot pilot?
A pilot should cover a clearly defined range of enquiries and a group of users whose questions will help assess the tool’s usefulness. Before launch, agree on the criteria for expanding access, including expected answer quality, acceptable response times and how errors will be handled. Users must be able to report a problem and contact support. During the pilot, the team should review conversations in line with the organisation’s data protection rules and collect customer feedback. It is important to check whether answers help users resolve their enquiries, including when questions are imprecise. The results help determine what needs to improve before the chatbot reaches a wider audience.
How can you assess whether a chatbot actually reduces customer support workload?
Use data from before the chatbot’s launch as a baseline: enquiry volumes for selected topics, handling times and how often customers make repeat contact. After implementation, compare the same measures while accounting for seasonality and changes in customer numbers. A user who does not contact an adviser after speaking to the chatbot has not necessarily resolved their issue, so customer feedback also matters. Support may receive fewer straightforward questions, leaving complex cases to account for a larger share of its work. In that situation, the average handling time per enquiry can rise even as the team’s overall workload falls. Considering these measures together provides a more reliable assessment of the business impact.
