TTMS Blog
TTMS experts about the IT world, the latest technologies and the solutions we implement.
Posts by: Marcin Kapuściński
AI in GxP Environments: From Pilot to Validated Deployment
Misclassifying a document can cause an important record to be filed in the wrong place within a system. In the pharmaceutical industry, such an error has implications for data integrity and the handling of regulated documentation. This was one of the examples discussed during the “Deploying AI in the GxP World” webinar, illustrating how many decisions need to be made before artificial intelligence can support day-to-day work in a GxP environment. On 13 October 2026, we attended the “Deploying AI in the GxP World” webinar, organised by ISPE Poland together with ProductLife Group (PLG). Naomi Ofili and Gabrièle Piaton from PLG spoke at the event. They discussed the risk assessment, validation and safeguards needed to move from an AI pilot to a deployment whose operation can be documented and assessed during an inspection. The webinar formed part of a series linked to the AI in Pharma 2026 conference. 1. What is GxP, and how does it affect AI deployment in pharma? GxP is an umbrella term for good practices covering areas such as medicinal product manufacturing, clinical trials and pharmacovigilance. One of the webinar’s central messages was that assessing an AI application should begin with its intended use and impact on a specific process. The same model can support tasks with very different implications for product quality, patient safety and documentation. The speakers illustrated this through translation. Translating internal correspondence and translating product information submitted to regulatory authorities require separate assessments. The same applies to literature analysis: the requirements depend on whether the search supports market monitoring or the identification of information about medicine safety. How the AI output is subsequently used also matters. Text, classifications or metadata may become part of regulated documentation. The assessment should therefore consider the solution’s role in the process, its impact on GxP records and the consequences of an error. The speakers recommended carrying out this analysis when developing the project’s business case, so that the validation scope, resources and costs can be identified early. 2. AI validation needs to be planned across the operational lifecycle Validation provides documented evidence that a system is fit for its intended use. The webinar repeatedly returned to the question of how to demonstrate this suitability for AI. The speakers referred to the risk-based approach described in GAMP 5 and to guidance on artificial intelligence. The materials cited included the ISPE GAMP Guide: Artificial Intelligence, published in 2025, which addresses AI-enabled systems in GxP areas. Another reference was the European Medicines Agency’s reflection paper on AI in the medicinal product lifecycle. The practical questions concerned data and evaluation criteria. Do the test examples reflect real documents and products? Was the test dataset independent of the data used to tune the solution? Which results will we consider acceptable, and did we define these criteria before testing began? The approach presented also covered limiting outputs unsupported by source material, assessing behavioural stability, monitoring performance, handling invalid inputs and detecting bias. Testing before release was combined with controls during operation and assessment of the impact of model changes. Reference collections of test cases developed with subject matter experts, described during the presentation as golden sets, can help. Revisiting the same cases across successive versions of a solution makes it possible to check whether a change has reduced performance on tasks that matter to the organisation. We explore practical ways to support testing and quality documentation in our comparison of AI QA tools for pharma. 3. Human oversight extends beyond approving an output Naomi Ofili distinguished between two levels of human involvement. The first is human-in-the-loop: a stage in the process at which an expert approves, edits or rejects an AI output. The second is oversight across the solution’s entire lifecycle, including monitoring, preventing uncritical acceptance of automated outputs and the ability to stop the system. In the workflow described, the AI output underwent automated checks before being reviewed by a designated expert. Information about the sources, instructions given to the model, its version and the approval was retained. This record made it possible to reconstruct the conditions under which the output was produced and identify who approved its use. The expert also helped define acceptance criteria and develop test cases. Their knowledge therefore informed the solution’s design and the way its quality was assessed. For a company planning a deployment, this means assigning responsibilities and allocating time for expert involvement from the outset. 4. What did the examples of literature monitoring, translation and data migration show? The speakers illustrated these principles with three examples from projects carried out by the PLG team. Each highlighted different requirements for assessing AI outputs and their subsequent use. 4.1. Literature monitoring for pharmacovigilance In the first example, AI supported the review of scientific publications for pharmacovigilance. The system proposed a category for each article and a rationale, both of which were reviewed by an expert. Gabrièle Piaton emphasised that a particularly challenging step was translating the specialists’ assessment method into a structured decision tree. This example showed the importance of documenting expert knowledge and classification criteria before automating a task. 4.2. Translation of regulated documentation The second example concerned AI-assisted translation reviewed by a human against quality criteria tailored to the risk associated with the document. The process also included reconstructing the formatting to facilitate the translators’ subsequent work. Assessment of the solution therefore covered content quality, the review process and the document’s suitability for further use. 4.3. Data migration and document classification In the third example, AI supported document classification, metadata extraction and mapping data to the target system’s structure. The speakers highlighted the difficulty of distinguishing between similar document classes and the consequences of misclassification. They stressed the need to trace data provenance, transformations and approvals, as well as to perform quantitative and qualitative verification before migrated data are put into use. 5. Hosting and costs should be considered before the pilot A separate part of the webinar focused on solution architecture. The speakers suggested deciding before the pilot where data would be processed, what information would be recorded in logs and whether the model version could be controlled. Their criteria for selecting an environment included confidentiality, data-processing location, update arrangements, the ability to reconstruct system activity and the cost of switching suppliers. The discussion also covered selecting models for individual tasks. Smaller, specialised models can help reduce costs if they meet the defined quality requirements. The proposed economic assessment focused on the cost of completing a task and obtaining the expected result. The speakers also highlighted the possibility of reusing components for information retrieval, testing, monitoring and output approval. We discuss the division of responsibilities between a pharmaceutical company and its supplier, along with update controls, in more detail in our article on SaaS validation in pharma and GxP requirements. 6. What helps move an AI pilot into day-to-day operation? Towards the end of the session, the speakers highlighted collaboration between technical teams, subject matter experts and the customer around a jointly agreed backlog. They also stressed the importance of appointing a business owner responsible for adoption of the solution and measurement of its outcomes. The experiences presented suggested a sequence of activities: defining the intended use and responsibilities, assessing risk, designing controls, building and validating the solution, and then maintaining it. Decisions about data quality, human oversight and change management affect every stage. For companies preparing a deployment, this means resolving several questions before the pilot: what task will AI perform, who will approve its outputs, how will their quality be assessed, and what will happen if the model or data source changes? The answers help teams plan the technical work, validation requirements and ongoing maintenance. 7. How does TTMS support deployments in pharmaceutical environments? Translating these decisions into requirements, tests and documentation requires collaboration between people who understand the business process, technology and quality assurance. At TTMS, we combine software development and maintenance for the pharmaceutical industry with TTMS Quality’s expertise in computerised system validation. Our validation services cover validation planning, requirements specification and review, test development and execution, results assessment and reporting. We use a risk-based approach and GAMP 5 guidance. The scope of work is tailored to the system’s intended use and the requirements of the specific process. If you are planning to use AI in a pharmaceutical process or preparing to take a pilot further, talk to our specialists about your solution’s requirements, risks and validation scope. Can an AI pilot in pharma start with synthetic data? Yes, synthetic data can help test an initial concept without using actual patient data or confidential documentation. However, they should reflect the structure, diversity and challenges of the intended application. Overly simple examples can give a misleading impression of the solution’s quality. It is worth including missing information, errors and ambiguous cases. The results of such a pilot are insufficient to demonstrate that the system is suitable for a real GxP process. Further assessment requires representative test data, agreed acceptance criteria and controls proportionate to risk. What should you do when AI and an expert assess the same document differently? The discrepancy should be reviewed according to an established procedure. The source document, AI output, solution version and the expert’s assessment and rationale should be retained. The cause may be a model error, ambiguous content or evaluation criteria that are too broad. Such cases help determine whether instructions, classification rules or tests need to change. An expert’s correction should not automatically trigger an update to a model used in a regulated process. Any change should be assessed, tested and approved, while retaining an independent dataset for quality evaluation. How should employees be prepared to use AI in pharmaceutical processes? Training should address the specific task, data and scope of the user’s responsibilities. Employees need to know what information they may submit to the system, how to check outputs and when to raise concerns. Exercises involving incorrect, incomplete or seemingly convincing responses are useful. They help assess whether users can verify an output rather than accept it uncritically. Instructions for handling failures or periods when the solution is unavailable are also helpful. Team preparation should include an assessment of practical skills, and training should be updated when the system or procedures change.
Read
Case study: How we improved a RAG chatbot for exam candidates
How can you help customers find a specific answer in hundreds of pages of documentation? An education company we work with planned to give its customers access to a chatbot that could answer questions about exam procedures. Before making the tool more widely available, the quality of its answers needed to improve. The TTMS team refined how the chatbot used the documents, increasing answer accuracy from around 70% to around 98% in an internal test. We prepared this case study using an internal presentation and project information shared by Jakub Kraśniewski, Senior AI Developer at TTMS. About the client: an education company supporting exam candidates Our client operates in the education sector and supports people preparing for a certification exam. Candidates need information at different stages, including registration, exam procedures, receiving results and appealing a grade. The company provided this information in PDF documents totalling several hundred pages, mainly in English. Finding the right answer therefore required customers to search through extensive materials themselves. The chatbot was intended to help them handle exam-related matters and reduce the support team’s workload. The challenge: improve answers before releasing the chatbot to customers Candidates needed to be able to describe their situation in their own words and receive clear guidance within around 4–5 seconds. Answers also needed to be complete. If a procedure applied only under certain conditions, the chatbot needed to explain them. TTMS was responsible for improving information retrieval in the existing solution, which had been built using CrewAI, a framework for developing AI agent-based systems. The chatbot used RAG (Retrieval-Augmented Generation) to generate answers from retrieved passages in the client’s documentation. Our analysis showed that the text chunks supplied to the model were too large, making them harder to match to a question. Text extraction from multi-column PDFs caused another problem: content from adjacent columns was being mixed together. Improving the chatbot therefore required examining what information the model received and whether it preserved the meaning of the source document. Solution and implementation: answers with the necessary context We changed how the documentation was divided into chunks, preserving the connections between sentences, paragraphs and pages. When the system found a sentence relevant to a question, it could also retrieve the surrounding paragraph or the entire page. This gave the model the information needed to explain a rule along with the conditions under which it applied. This approach allowed us to use the client’s existing materials and retain classic RAG. According to Jakub, the additional storage and processing costs were acceptable for this volume of documentation. We evaluated the changes using an internal set of questions covering the full scope of the materials. We also included real user questions collected anonymously during an earlier prototype launch. This helped us assess answers to questions phrased in the way customers actually asked them. We explain the technical approach in our article “RAG for chatbots using CrewAI – notes from a TTMS Tech Talk”. Results: around 98% answer accuracy in an internal test According to the data shared by Jakub Kraśniewski, answer accuracy increased from around 70% to around 98%, an improvement of approximately 28 percentage points. The solution also maintained a fast response time. This result applies to the internal test set prepared for the project. It confirmed the improvement in answer quality and helped prepare the tool for the next stage: release to a subset of customers. The actual reduction in the support team’s workload had not yet been assessed. What comes next? Evaluating the chatbot in everyday candidate support After completing internal testing, the company planned to make the chatbot available to a subset of customers. This stage will help establish whether candidates receive enough assistance to resolve their enquiries and need to contact a support adviser less often. The assessment will draw on chatbot usage data and support enquiries. Let’s talk about a chatbot for your company Would you like to make company documentation easier for customers or employees to use? Or do you already have a chatbot whose answers need improvement? Talk to the TTMS team. We will review your materials and users’ needs to recommend a solution that fits your business. Where should you start with a customer service chatbot project? Start by reviewing support enquiries to identify the questions customers ask repeatedly. Issues for which the company has clear, documented answers are good candidates for the initial scope. Next, define who the chatbot should help and what users should be able to accomplish through a conversation. This makes it easier to set the scope of a pilot and decide how to assess its results. Involve customer service staff early, as they understand common misunderstandings and situations that require an individual decision. A scope prepared this way gives the implementation team a concrete starting point. Can you improve an existing chatbot without rebuilding it from scratch? In many cases, individual components of an existing solution can be improved, although the extent of the changes requires an initial assessment. The team should review examples of incorrect answers and identify where the problem occurs. The cause may lie in document preparation, information retrieval or how the model uses the retrieved content. The limitations of the current platform and access to its configuration also matter. Before deciding to rebuild, it is worth comparing the effects of selected changes using the same set of questions. This helps establish whether further development of the existing tool is justified. What determines the cost of implementing a chatbot that uses company documents? The cost depends partly on the condition of the source materials, the integrations required and the tool’s operating requirements. Documents with complex layouts or inconsistent content may need more preparation. Connecting the chatbot to company systems is a separate task, particularly when access to information depends on user permissions. The budget should also cover testing, support after launch and knowledge base updates. Ongoing costs depend on conversation volumes, the models selected and how questions are processed. An estimate should therefore distinguish between the cost of building the solution and the expected cost of maintaining it. How do you plan a customer chatbot pilot? A pilot should cover a clearly defined range of enquiries and a group of users whose questions will help assess the tool’s usefulness. Before launch, agree on the criteria for expanding access, including expected answer quality, acceptable response times and how errors will be handled. Users must be able to report a problem and contact support. During the pilot, the team should review conversations in line with the organisation’s data protection rules and collect customer feedback. It is important to check whether answers help users resolve their enquiries, including when questions are imprecise. The results help determine what needs to improve before the chatbot reaches a wider audience. How can you assess whether a chatbot actually reduces customer support workload? Use data from before the chatbot’s launch as a baseline: enquiry volumes for selected topics, handling times and how often customers make repeat contact. After implementation, compare the same measures while accounting for seasonality and changes in customer numbers. A user who does not contact an adviser after speaking to the chatbot has not necessarily resolved their issue, so customer feedback also matters. Support may receive fewer straightforward questions, leaving complex cases to account for a larger share of its work. In that situation, the average handling time per enquiry can rise even as the team’s overall workload falls. Considering these measures together provides a more reliable assessment of the business impact.
Read
GPT-6.1 Sol for business: Greater capabilities, lower costs
On 29 September 2026, at DevDay, OpenAI introduced GPT-6.1 Sol, just one week after the launch of GPT-6 Sol. The company aimed to bring Sol closer to GPT-6 Astra’s capabilities in coding, document analysis and application use, while keeping the model’s existing base prices. OpenAI kept GPT-6 Sol’s pricing: USD 2 per million input tokens and USD 10 per million output tokens, one fifth of Astra’s standard rates. OpenAI also links the update to the development of AI agents, programs that complete tasks step by step using available tools. Analysing documentation, preparing a code change and checking that it works may require multiple calls to the model. The cost of this work grows with the number of operations. GPT-6.1 Sol is intended to let application developers tackle more demanding tasks within the same budget. 1. What is GPT-6.1 Sol, and what has changed since GPT-6 Sol? GPT-6.1 Sol builds on GPT-6 Sol and is designed for complex tasks involving code, documents and applications. OpenAI’s documentation also lists support for tools, including search and code execution. The model can be used in an AI agent connected to a company’s data sources and tools. An agent could read a customer enquiry, find the relevant specification, compare requirements and draft a response. The company must define which data the agent can access, which actions it can perform independently and which require an employee’s approval. Base prices for input and output tokens remain the same as for GPT-6 Sol. The rate for reusing cached data, however, falls from USD 0.20 to USD 0.10 per million tokens. Companies using GPT-6 Sol should test both versions on the same tasks and compare the quality of their results, the number of retries and token usage. 2. What do benchmark results tell us about business applications? OpenAI publishes benchmark results covering document analysis and multistep tasks. When comparing results, check the reasoning level used. A higher level allows the model to spend more computation on analysing a task, which may increase token usage. Benchmark Area assessed Result reported by OpenAI GDP.pdf Answering questions based on complex PDF documents. Results close to Astra’s at approximately one fifth of the cost per task. AutomationBench 1.0.6 Multistep tasks in business applications. A lead of 4.8 percentage points over GPT-6 Sol at the medium reasoning level. OSWorld 2.0 Using desktop applications. A score 2.1 percentage points below Astra’s. The cost per task is approximately one seventh of Astra’s. Both models were tested at the maximum reasoning level. Each benchmark assesses different model capabilities, so their results should be considered separately. GDP.pdf, developed by Surge AI, tests whether a model can analyse workplace documents and answer questions about their contents. The model must also read tables, diagrams and footnotes correctly and connect information from different parts of a document. The result therefore shows how it handles materials similar to those reviewed by procurement departments or technical teams. Zapier’s AutomationBench assesses task execution in simulated applications. It checks whether the model actually made the required changes, such as updating the correct record and completing every requested action. This matters for businesses: a model’s claim that it has completed a task should be supported by data saved in the system. OSWorld tests computer use, including actions performed through application interfaces. The result cited by OpenAI includes credit for partial task completion. A high score may therefore mean that the model completed most steps correctly but missed the final one, such as submitting a form. When assessing a model’s suitability for automation, companies should also check how often it completes the entire task. These benchmark results help companies choose use cases for a pilot. Testing the model on their own documents and in the applications they use will show whether it meets their requirements and how much time is needed to check and correct its work. 3. How much does GPT-6.1 Sol cost, and how should you read the pricing table? API usage, which allows an application to access the model, is billed in tokens: units of processed text that often correspond to parts of words. Input includes instructions and supplied materials. Billable output tokens also include reasoning tokens, which are not visible in the response. Rates in USD per million tokens in Standard mode, for requests containing up to 272,000 input tokens: Model Input Cache read Cache write Output GPT-6 Sol 2.00 0.20 2.50 10.00 GPT-6.1 Sol 2.00 0.10 2.50 10.00 GPT-6 Astra 10.00 1.00 12.50 50.00 This comparison covers standard processing. Faster processing modes and requests with more input data have separate rates, and processing in a selected region may incur a surcharge. Caching can reduce the cost of subsequent requests that use the same instructions or documents. Savings depend on how much data can be reused. The total cost of a task includes token usage, caching charges and the tools used. 4. Example: how much could you save on 1,000 tasks per month? Let’s work through an example: a company completes 1,000 tasks per month, each using 20,000 input tokens and 3,000 billable output tokens, including reasoning. We assume identical usage for both models. To keep the calculation clear, we exclude cache reads and writes, tool fees and surcharges. Token usage must be measured in an actual deployment. Monthly cost component GPT-6.1 Sol GPT-6 Astra 20 million input tokens USD 40 USD 200 3 million output tokens USD 30 USD 150 Total token cost USD 70 USD 350 Token savings amount to USD 280 per month. Now let’s add another assumption: an hour of work by the person checking the results costs the company USD 30. If checking Sol’s results took an average of one extra minute per task, monthly labour costs would rise by USD 500. In this example, using Sol would cost the company USD 220 more per month overall than using Astra. If checking Sol’s results takes the same amount of time as checking Astra’s, or less, lower token spending may reduce the total cost of completing a task. The pilot should compare result quality, model costs and the time spent checking outputs. To calculate the cost per successfully completed task, divide the total process costs by the number of tasks completed according to the agreed requirements. The calculation should also include failed attempts, corrections, exception handling and maintenance. Data preparation and system integration costs should be spread over the planned period of use. 5. Which GPT-6.1 Sol use cases are worth testing in your company? A pilot could cover comparing documents and proposals, preparing information for customer service and developing code fixes. 5.1 Comparing technical documents and proposals A procurement department receives proposals containing specifications, parameter tables and service terms. The system could prepare a comparison against an agreed set of requirements. For each value, it should identify the source document and the location where it found the information, and flag any missing data. The test set should include two versions of the same specification, parameters expressed in different units and a qualification stated in a footnote. The reviewer can then check whether the system selected the current version and interpreted the qualifications correctly. The total time needed to prepare and check the comparison should be compared with the time needed to complete the task without AI. 5.2 Preparing customer service information across multiple systems After receiving a sales enquiry, an agent could find the customer’s account history, check the status of previous support tickets and prepare an entry in a customer relationship management (CRM) system. This requires connections to the relevant applications and access to the necessary data. The first stage of deployment could end with proposed changes being presented to an employee. Once approved, the system saves them and checks the result. Tests should include records for different customers with similar names to check whether the agent selects the correct one. The same task should also be assigned again to check whether the agent recognises the existing entry and avoids creating a duplicate. 5.3 Diagnosing bugs and preparing software changes An IT team can compare the models when analysing tickets, reproducing bugs and preparing fixes with tests. Previously resolved tickets, where the cause of the bug and the required software behaviour are known, make useful test cases. The assessment should cover the correctness of the change, test results and the time needed for a developer’s review. It should also check whether the model limited its modifications to the necessary scope. Too many code changes can make the review longer and make it harder to identify which changes fix the reported bug. 6. How can you compare GPT-6.1 Sol with Astra in your own process? OpenAI’s model selection guidance recommends comparing Sol and Astra on the same task. The comparison should cover result quality, completion time and task cost under conditions close to the intended use. Choose a task with a clear completion criterion. This could be comparing all required proposal parameters or preparing a code change that passes specified tests. Build a set of examples from day-to-day work. Include typical cases, incomplete data and known exceptions. Keep the materials used to refine instructions separate from those used to evaluate the final configuration. Ensure comparable conditions. Give the models the same materials, tools and permissions. Record reasoning settings and limits. You can then adjust each model’s settings to meet the same quality requirements. Calculate the full cost and check the result. Measure correctness, retries, system processing time and human review time. Also record cases that require manual completion. Expand use once the agreed conditions are met. Define which results can be used immediately, which require approval and when a case should be referred to a specialist. OpenAI recommends testing AI systems on tasks that reflect day-to-day work and having their results assessed by people with relevant expertise. Tests should be repeated after changes to the model, instructions or data sources. You can also test a division of work between the models. Sol would handle cases where it meets the required quality standard, while Astra would handle selected, more difficult cases. Define in advance when a task should be passed to Astra, for example when Sol misinterprets documents or produces an incorrect response. If required data is missing, it should be supplied before another analysis. The cost of that additional analysis must be included in the calculation. The final division of work will depend on the test results. 7. Let’s discuss automation in your company At TTMS, we help select AI solutions for specific processes and integrate them with business systems. If you are considering GPT-6.1 Sol, contact us. Together, we can identify tasks worth including in a pilot and determine how to assess result quality and the full cost of completing them. Can GPT-6.1 Sol be connected to a company knowledge base? Yes. An application can provide the model with information retrieved from approved company materials. This can be done using retrieval mechanisms described in OpenAI’s documentation. The approach in which a system first retrieves relevant source passages and then uses them to prepare an answer is known as retrieval-augmented generation (RAG). Implementation requires keeping documents up to date, identifying versions and respecting user access permissions. Providing sources helps users check the answers. Performance should also be assessed on questions for which the knowledge base does not contain enough information. Does switching from GPT-6 Sol to GPT-6.1 Sol require changes to the application? The scope of changes depends on how the application is integrated. The team should check the available interfaces, reasoning settings, tool support and response formats. For example, the GPT-6.1 Sol model documentation states that tool use requires the Responses API and that the none reasoning setting is not supported. This may affect applications built for the previous version. After adjusting the configuration, the team should test it against a saved set of tasks, including error handling. It is worth retaining the option to revert to the previous configuration if the new one does not meet requirements. What is GPT-6.1 Sol? GPT-6.1 Sol is an OpenAI model designed for tasks involving code, documents and business applications. It builds on GPT-6 Sol while keeping the same base prices for input and output tokens. It also supports tools, allowing developers to use it in AI agents that retrieve information and perform actions across connected systems. For businesses, its appeal lies in the balance between capability and cost. Its suitability for a particular process should be assessed through tests that measure result quality, token usage and the time needed for human review.
Read
GPT-6 Cyber: What OpenAI’s Cybersecurity Model Means for Business
OpenAI has introduced GPT-6 Cyber, an AI model designed for cybersecurity work. For companies managing customer portals, payment services, and internal platforms, potential benefits include faster vulnerability investigations, more frequent security reviews, and better support for engineers preparing and testing fixes. Its practical value will depend on whether it helps teams confirm vulnerabilities and test fixes while reducing the time specialists spend on those tasks. What Is GPT-6 Cyber? GPT-6 Cyber is positioned as OpenAI’s specialized model for cybersecurity work. Before its introduction, BusinessToday reported that selected customers were testing an alpha version through Daybreak Red. According to Investing.com, citing Fortune, OpenAI is also developing a separate product to help customers deploy GPT-6 Cyber securely, automate security tasks, and patch vulnerabilities. The product would give OpenAI greater oversight of how its models are used. For businesses, this raises practical questions about how the service would fit into existing security operations: which tasks it could carry out, which actions would require approval, and what information OpenAI would receive about its use. These details would influence both implementation costs and the suitability of the service for handling sensitive company systems. There is already a documented foundation for this approach. OpenAI describes GPT-5.6 Cyber as a model for approved users conducting advanced vulnerability research and security testing. It also offers Codex Security, an application security agent that helps teams investigate vulnerabilities and prepare fixes. These products show how OpenAI already supports security work; GPT-6 Cyber needs its own assessment of capabilities and performance. For a buyer, three parts of an AI security solution deserve separate attention: Part of the solution What it determines What the business should verify Model The analysis and reasoning available for a task Accuracy on relevant security cases, limitations, and cost Access program Who can use particular capabilities and under which conditions Eligibility, approval requirements, and permitted activities Application and integrations The data the system can inspect and the actions it can execute Supported tools, permissions, review controls, and audit records Before purchasing access, check how the solution will connect to the company’s code repositories, monitoring tools, and process for approving changes. Why Cybersecurity AI Matters to Business Leaders Security work competes for engineering capacity. Investigating a suspicious code change, checking a supplier’s advisory, and validating a patch all draw on people who also maintain applications and deliver new features. AI creates an opportunity to move some of the research and preparation into a repeatable workflow, allowing specialists to spend more time on judgment, validation, and decisions about production systems. Faster security work can also help companies keep their services running. A retailer may need to secure a checkout integration without disrupting sales. A manufacturer may need to understand the consequences of updating software connected to operational processes. A financial services company may need clear evidence of what was investigated and how a problem was resolved. In each case, useful automation must fit the business process surrounding the software. OpenAI’s broader investment shows the importance it assigns to this market. On September 3, 2026, the company announced a $1 billion commitment covering subsidized Daybreak access, training, technical support, and partnerships for organizations protecting essential services. Four Business Uses to Evaluate for GPT-6 Cyber The following scenarios are our analysis of where a cybersecurity model could create value. They are proposed evaluation areas whose suitability for GPT-6 Cyber requires testing. Each depends on the capabilities, tools, and permissions available in the eventual deployment. 1. Investigating Vulnerabilities in Business Applications A useful pilot could focus on a small set of applications with a clear business owner. The model would be given approved code and architecture information, then assessed on whether it helps explain suspected weaknesses and identify the evidence needed to confirm them. For example, a team might investigate whether a customer portal consistently enforces access permissions across related services. The business benefit would come from reducing the time needed to reach a defensible conclusion. A helpful result should identify affected components, explain the conditions under which the issue matters, and make it easier for an engineer to reproduce the problem in an authorized test environment. 2. Preparing and Testing Security Fixes Once a vulnerability is confirmed, the next task is to prepare a change that addresses it. A security AI workflow could help draft a patch, identify related code paths, and propose tests for both the security issue and normal application behavior. Existing Codex Security documentation already describes workflows for reviewing changes, validating findings, and preparing fixes, making this a concrete area for evaluating future model improvements. For the business, success means less engineering effort per validated fix while maintaining release quality. Engineers should still review the change and run appropriate tests before it reaches production. 3. Prioritizing Work Using Business Context Using an up-to-date list of systems, their owners, and supporting documentation, an AI system could help connect technical findings with business consequences. This could help the team decide which vulnerabilities to fix first and who should handle them. 4. Helping Teams Investigate Security Incidents Another use to test is building an incident timeline from security alerts and system logs, with internal procedures guiding the investigation. An evaluation could test whether the model links its statements to evidence, distinguishes observations from hypotheses, and identifies missing information. This scenario would require suitable integrations; it should not be assumed to be a built-in GPT-6 Cyber feature. A useful output could help the next analyst continue an investigation or help a service owner understand the affected business process. Decisions such as disabling accounts, blocking traffic, or isolating systems need explicitly assigned authority because they can interrupt legitimate activity. What Could This Look Like in a Customer Portal? Consider a company preparing a new version of a customer portal connected to its order management system. A security review flags a possible weakness in how the portal checks access to order details. In a controlled pilot, an AI system would receive the relevant code, a description of expected permissions, and test accounts containing fictional customer data. The team would assess whether it can help trace the affected logic, produce a clear explanation, and propose a test that demonstrates the problem. If the finding is confirmed, engineers could evaluate its suggested correction and additional regression tests, which check that existing functions still work after a change. The final result should be a reviewed change with evidence: what was wrong, what was modified, which tests passed, and who approved the release. This would require access to the relevant code, a test environment, and a process for assigning confirmed issues to engineers. How to Measure the Business Value of GPT-6 Cyber A pilot should compare the AI-assisted workflow with the team’s current process using comparable cases. Include confirmed vulnerabilities, issues previously dismissed as false alarms, and cases where the evidence is incomplete. Record analyst review time as well as model execution time. An answer produced quickly can still require substantial investigation. The following measures provide a practical basis for a decision: Measure What to record Why it matters Time to assess a suspected vulnerability Elapsed time and analyst effort needed to confirm or dismiss an issue Shows whether the workflow accelerates investigation Finding quality Confirmed findings, false alarms, and missed issues in a reference test set Reveals whether apparent productivity comes with additional errors Time to a tested and approved fix Time from confirmation to a tested, approved correction Connects AI assistance to remediation Engineering effort Hours spent reviewing, correcting, testing, and documenting outputs Makes supervision costs visible Total cost per resolved case Model use, tools, test environments, integration, and staff time Supports a realistic decision about scaling For example, if a pilot saves analyst time but generates extra work for developers, both effects belong in the assessment. If it uncovers more valid vulnerabilities, the organization also needs capacity to fix them. Access, Pricing, and Deployment Questions for Buyers OpenAI’s existing Daybreak documentation requires appropriate organization approval and project access. It distinguishes access to a program from access to a particular model. For GPT-6 Cyber, organizations should verify the applicable requirements directly, including whether access is limited to an evaluation or supports the planned production use. Commercial evaluation should cover pricing, usage limits, supported integrations, and the handling of company data. Teams should establish what source code, logs, and configuration information would be processed; where processing occurs; how long data is retained; and which contractual controls apply. Availability through a particular cloud provider or within an existing subscription should be confirmed for the specific offering. A pilot budget should include implementation and review costs alongside any model usage fees. How to Keep AI Security Work Under Control A business should define the scope of an AI security workflow as carefully as it defines the scope of an external security assessment. Specify the systems it may inspect, the information it may access, and the actions it may take. Separate permission to analyze a problem from permission to change a live service. OpenAI’s published Daybreak guidance recommends isolated environments, monitoring of agent actions, and enforced limits on authorized activity. For an initial business pilot, that supports a controlled test environment, narrowly scoped credentials, review of proposed changes, and records that allow the team to understand what happened. The operating process also needs an owner. Someone must review unresolved findings, decide when additional evidence is required, and stop a workflow that behaves unexpectedly. Training should cover how to challenge an AI-generated conclusion and how to recognize when the system lacks enough information to proceed. Where Businesses Should Start Start with one application and one recurring security task, such as investigating suspected vulnerabilities or testing fixes. Assign someone to review the results and agree how you will measure accuracy, time saved, and the effort required from engineers. A focused pilot can help establish whether the approach is useful enough to expand. Considering AI for your company’s security processes? Talk to TTMS about the task you want to improve, the systems involved, and your data requirements. Together, we can explore a suitable approach and define what a useful pilot should demonstrate. Discuss your AI use case with TTMS How is GPT-6 Cyber different from using a general-purpose AI model for cybersecurity? GPT-6 Cyber is described in published reports as a model focused on cybersecurity, with early testing through OpenAI’s Daybreak Red program. General-purpose models can also assist with security tasks, such as explaining code, reviewing documentation, and analyzing supplied findings. The question for a business is whether the specialist model produces more accurate, useful results on the tasks its team actually performs. Its name alone does not establish that advantage. A meaningful comparison should give both models equivalent information, tools, and time, then have security specialists assess the results. Published evaluations can help inform that comparison, but results from another model should not be attributed to GPT-6 Cyber. Can GPT-6 Cyber replace a cybersecurity team or an external security provider? A company should retain qualified people responsible for assessing risks, validating findings, and approving changes. A model’s analysis depends on the information and tools available to it, which may leave gaps in its understanding of the company’s systems. Security specialists also consider operational priorities, investigate ambiguous evidence, and coordinate the response when something goes wrong. AI assistance may reduce the effort needed for particular tasks, but that needs to be demonstrated in practice. For a business using an external security provider, a useful discussion is how the provider validates AI-generated work and whether any time savings improve the service. Responsibility for protecting systems and resolving problems should remain clearly assigned. Is GPT-6 Cyber useful for a company that mainly uses software from external vendors? Its usefulness would depend on what the company controls and what information it can access. A business using standard cloud applications may have limited visibility into the vendor’s underlying code. Its own responsibilities may include account permissions, configuration, integrations, and custom extensions. Those areas could provide relevant tasks for AI-assisted analysis, subject to the model’s verified capabilities and the available integrations. Before testing, establish which systems the company is authorized to assess and what requires the vendor’s involvement. Findings affecting the vendor’s product should be reported through its security process so that they can be investigated and addressed. Does an AI security review that finds no vulnerabilities mean an application is secure? No. A review can miss vulnerabilities because of incomplete information, limited test coverage, or errors in the analysis. A finding-free report should therefore describe what was examined, which tests were performed, and what remained outside the scope. For example, reviewing selected source files does not establish that the application’s production configuration and access permissions were also checked. The result should be considered alongside other security evidence, including testing and specialist review appropriate to the application’s risk. Record unresolved questions and limitations so that a clean report does not create unjustified confidence. How should a company assess a supplier offering “GPT-6 Cyber-powered” services? Ask the supplier to explain which model it uses, how it accesses that model, and which parts of the service rely on it. Request a demonstration using a representative, authorized task and ask to see the evidence supporting the findings. Establish who reviews the output, who implements corrections, and what happens when the analysis is wrong or incomplete. The service description should also explain how company data is handled and which actions require approval. If the supplier changes the underlying model, ask how it checks that the service still meets the agreed requirements. Evaluate the offer against clear deliverables, such as validated findings and tested fixes, with named people responsible for the work.
Read
RAG for Chatbots using CrewAI: Notes from a TTMS Tech Talk
Tech Talk is an internal series of technology sessions for TTMS employees, where we share knowledge and project experience. We demonstrate tried-and-tested tools, discuss challenges we have encountered and explain the solutions that have helped us in our work. Topics include artificial intelligence, data analytics, Salesforce, AEM and project management. Presentations are followed by time for questions, discussion and sharing ideas. During the session “RAG for Chatbots Using CrewAI”, held on 17 September, Jakub Kraśniewski, Senior AI Developer at TTMS, discussed improvements to a chatbot using a client’s documentation. He presented the challenges involved in preparing and retrieving information, the solution implemented and the approach to evaluating answer quality. The project involved a company in the education sector whose customers were preparing for a certification exam. The chatbot was intended to help them find information about registration, exam procedures, grading and appeals, thereby reducing the support team’s workload. It used several hundred pages of publicly available PDF documents, mainly in English. The team needed a way to retrieve relevant information from these materials while meeting a response time requirement of around 4 to 5 seconds. 1. How does RAG help a chatbot use company knowledge? Jakub began the presentation by explaining how RAG (Retrieval-Augmented Generation), a method of generating answers using retrieved source material, works. The system finds information relevant to the user’s question and passes it to a language model as context for the answer. In this project, the retrieved material consisted of passages from documentation describing exam rules and procedures. After extracting text from the documents, the system divides it into smaller chunks. An embedding model (an AI model that represents semantic features of text as numbers) converts these chunks into vectors stored in a database. The user’s question is processed in the same way. Comparing these representations allows the system to retrieve passages that are semantically related to the question. Jakub emphasised that the team is responsible for the quality of the material passed to the model. This involves checking whether the text was extracted correctly, whether the chunking preserved the necessary context and whether retrieval provides information useful for answering the question. The challenges the team encountered in the CrewAI-based solution demonstrated the importance of these steps. 2. What made information retrieval difficult in the CrewAI project? After explaining the basics of RAG, Jakub shared his experience from a project using CrewAI, a framework for building AI agent-based systems. He discussed three problems encountered in the configuration used: overly large text chunks, the absence of an additional relevance assessment and errors in PDF text extraction. 2.1 Overly large document chunks In the configuration Jakub described, text was split into chunks of 4,000 characters. The system retrieved five such chunks for each question, passing up to approximately 20,000 characters of source material to the model. A large chunk can contain information on several different topics, making it harder to match it to a specific question. The model generating the answer must then select the relevant information from the supplied content. In this project, the chunking approach therefore needed to be adapted to the structure of the documents and users’ questions. 2.2 No additional assessment of search result relevance Jakub pointed out that the configuration lacked reranking, which involves reassessing and reordering search results according to their usefulness for answering the user’s question. The system can first retrieve a larger number of passages, then assess them further to select those most useful for preparing an answer. Jakub presented this method as a potential improvement whose value should be evaluated by checking both answer quality and response time. 2.3 Incorrect text reading order in multi-column PDFs Another problem involved document text extraction. The tool read multi-column PDFs row by row, merging content from adjacent columns. This disrupted the order of sentences and made subsequent information retrieval more difficult. The resulting text was then split into chunks. The error therefore originated during data preparation and affected the subsequent stages of document processing. This example showed why evaluating RAG quality should begin with comparing the extracted text against the source document. 3. How does response time affect the choice between Classic RAG, Agentic RAG and Graph RAG? A key project requirement was a response time of around 4 to 5 seconds. Jakub discussed three RAG approaches in terms of data preparation costs, the ability to evaluate their operation and the time needed to handle a question. Approach How it works, as discussed during the session What to consider when choosing Classic RAG Retrieves passages from a knowledge base, optionally reranks them and passes the context to the model. Document chunking quality, retrieval relevance and the amount of context provided. Agentic RAG An agent selects tools and a retrieval method, running additional queries as needed. The ability to adapt retrieval to the question, along with the time and cost of additional operations. Graph RAG Retrieval uses a knowledge graph describing entities found in the source material and the relationships between them. The effort required to build and maintain the graph, and how useful the relationships are for answering users’ questions. In an agentic approach, the model can use several tools, such as vector search, keyword search or filtering by metadata describing the document. Additional steps allow the system to expand its search for information, while their number and sequence affect response time. In the graph-based approach presented, some of the work takes place when building the knowledge base. Entities and the relationships between them are extracted from the text. This mechanism also underpins GraphRAG as described by Microsoft. Jakub highlighted the costs of this preparation and the difficulty of manually analysing a complex graph. In this project, the response time requirement favoured further development of classic RAG. The team focused on document chunking and context selection. 4. How does hierarchical document chunking help preserve context? The solution organised the material into three connected levels: pages, paragraphs and sentences. The system retained information about which paragraph each sentence belonged to and which page contained that paragraph. Content was represented in the vector database at different levels of detail. This allowed retrieval to identify both individual sentences and larger passages containing the required information. According to Jakub, the additional cost of storing and processing these representations was acceptable given the volume of material in the project. Finding a relevant sentence made it possible to retrieve its entire paragraph and provide the model with broader context. The system could also retrieve the whole page when needed. Suppose a user asks about the deadline for appealing an exam result. The system finds a sentence specifying the deadline, then retrieves the entire paragraph explaining when the appeal period begins and how to submit an appeal. This allows the model to account for these conditions in its answer. 5. How can you evaluate RAG quality using your own data? In the final part of the presentation, Jakub emphasised the importance of a benchmark, a set of tests used to compare different versions of a solution. He discussed checking retrieval results against information labelled by a human and using a language model to evaluate answers. In practice, it is useful to assess two stages separately. The first concerns retrieval: did the system return a passage containing the required information? The second concerns the answer: did the model use the supplied material correctly? This distinction helps identify which stage needs improvement. In additional information shared after the session, Jakub clarified the testing method and results. The test set included questions covering the full scope of the documentation, along with real user questions collected anonymously during a prototype launch at the beginning of the year. Answer accuracy increased from around 70% to around 98%, an improvement of approximately 28 percentage points. This result applies to the internal test conducted in this project. According to Jakub, the solution also maintained a fast response time. When he shared these details, the chatbot had completed internal testing, and the company planned to make it available to a subset of customers. Reducing the support team’s workload and making information easier to access remained deployment goals. Assessing whether those goals have been achieved requires data from actual use. The embedding model is another component to evaluate. Jakub noted that its selection should take into account the language of the source material and its performance on the team’s own dataset. The choice of this model affects which passages the system retrieves before it begins generating an answer. 6. What can companies implementing a chatbot learn from this experience? The project shows how specific requirements guide RAG development. The expected response time helped narrow down the choice of solution, document analysis revealed problems with text extraction and chunking, and an internal test made it possible to assess the impact of the changes. When planning a similar implementation, it is worth addressing five areas: Source material: check whether document text extraction preserves meaning and reading order. Document chunking: adapt chunk size and the connections between chunks to the structure of the material. User questions: prepare a test set that reflects the tasks the chatbot is intended to support. Response time: establish expectations and account for them when comparing approaches. Quality assessment: check both the relevance of the retrieved information and how it is used in the answer. Let’s talk about AI in your company TTMS is home to experts who, like Jakub, combine technical knowledge with experience from client projects. During Tech Talks, they share solutions tested in practice and apply what they have learned to subsequent implementations. Are you planning a chatbot that uses your company’s documentation, or looking to improve the answer quality of an existing tool? Let’s talk. We will review your materials, users’ needs and business goals to recommend an appropriate way to use AI. Contact the TTMS team! What documents can a RAG chatbot use as a knowledge base? A RAG chatbot can use company policies, product manuals, procedures, FAQs and other materials containing information relevant to its users. Sources may include PDFs, Word documents, website content and knowledge base articles, depending on the integrations available. Scanned documents require optical character recognition (OCR) to turn images of text into searchable content. Tables, diagrams and complex layouts may need additional processing to preserve their meaning. Before adding documents, check that they are accurate, current and approved for the intended audience. Clearly structured materials help the system retrieve information and provide useful context for its answers. How do you keep a RAG chatbot’s knowledge base up to date? Keeping a RAG chatbot up to date requires a process for detecting and processing changes in its source materials. Depending on business needs, updates can run on a schedule or be triggered when a document is added, edited or removed. The system then updates the searchable content and its associated representations, such as embeddings. Version information and effective dates help distinguish current guidance from older material. Deleted or superseded documents should also be removed from active search results, and cached answers may need refreshing. Assigning an owner to each content area helps ensure that someone remains responsible for the information the chatbot uses. Can a RAG chatbot provide sources for its answers? Yes, a RAG chatbot can include links, document titles, page numbers or quoted passages alongside its answers. This requires the system to preserve source information when processing documents and connect retrieved passages to the response. Useful citations let users open the relevant material and check the context for themselves. The system should also verify that each citation supports the claim it accompanies. A source link alone provides no guarantee that an answer accurately reflects the document. During testing, teams should check both answer quality and citation accuracy, including whether users can access the referenced material. How can a RAG chatbot respect access permissions for company documents? A RAG chatbot can use the signed-in user’s identity and access rights to determine which documents it may retrieve. Permission checks should happen before restricted content reaches the language model. The same controls need to cover document previews, citations and any cached responses that contain protected information. When access rights change in a source system, those changes must also be reflected in the chatbot’s retrieval process. Teams should test the solution using accounts with different roles, including users with limited access. These checks help confirm that each person receives answers based on information they are authorised to view. What should a RAG chatbot do when it cannot find an answer? When the available documents provide insufficient information, a RAG chatbot should clearly explain that it cannot answer reliably from its sources. It can ask a clarifying question if the request is ambiguous or suggest a related document that may help. For questions requiring further assistance, it can direct the user to the appropriate team or support channel. The system needs explicit rules for handling incomplete, conflicting or missing information. Testing should include questions whose answers are absent from the knowledge base, so the team can assess this behaviour. Reviewing unanswered questions can also reveal gaps in company documentation and priorities for future updates.
Read
ChatGPT for financial services: what does combining GPT with professional data sources offer?
On 10 September 2026, OpenAI announced ChatGPT for Financial Services, a solution combining GPT-6 Astra with professional financial data sources and tools for preparing analyses. The product was developed in collaboration with Morgan Stanley and Evercore. It is designed for financial institutions, with an initial focus on investment banking and equity research. It allows analyst teams to find data, perform calculations and prepare client materials in one place. This could reduce the time spent gathering information and transferring it between tools. In this article, you will learn: what data and features ChatGPT for Financial Services offers, what preparing a company analysis with GPT could look like, why metric calculations and data sources need to be checked, which stages require an analyst’s review, how to assess whether implementation is worthwhile for your company. How does ChatGPT for Financial Services support analysts? Materials published by OpenAI and its data providers describe several specific use cases: Comparing companies. Daloopa, a provider of company financial data, makes selected data and metrics available for comparing business performance. Source references help analysts verify where the figures come from. Finding companies that meet specific criteria. Daloopa also describes searching for companies by business activity or geographical region. The resulting list can provide a starting point for further market analysis. Preparing client materials. OpenAI describes creating valuation models, research notes and presentations using company templates for Excel, Word and PowerPoint. ChatGPT for Financial Services provides access to selected data from Daloopa, PitchBook, LSEG News and Crunchbase. OpenAI is also developing integrations that will allow institutions to use data covered by their existing subscriptions. These include S&P Capital IQ, LSEG, MSCI, Dow Jones Factiva and Moody’s. From data to company analysis: five steps in the workflow Let’s walk through preparing a comparison of two industrial companies ahead of a client meeting. The analyst needs to assess profitability, explain material differences and prepare a short note with a results table. Using fictional data, we will show what to check during a pilot, from selecting information to approving the final material. 1. Defining the question and scope of the comparison First, we establish which period to compare: the last full year, a six-month period or the trailing twelve months. We also check whether the figures cover the entire corporate group or an individual company, which currency they use and how each metric was calculated. In our example, we use consolidated data for both groups for the same calendar year. Amounts are stated in millions of euros. Before comparing results, we need to check the start and end dates of the reporting periods. One company’s financial year may end in December, while another’s ends in March. The documentation for the US SEC’s EDGAR database also highlights these differences. The analyst must then decide how to account for the mismatch and whether additional data is needed. The agreed approach should be recorded in the instructions for the AI and included with the completed analysis. This gives the model clear guidance and helps the reviewer understand which data was compared and why. 2. Gathering data and identifying its sources For each important figure, record the company and period it relates to, the units used and how it was calculated. A reference to the specific table or explanatory note in the report is also needed. Keeping the source document and its retrieval date makes it easier to review or update the analysis later. According to OpenAI’s description, ChatGPT for Financial Services lets users locate specific tables and document passages, highlighting the information used in the analysis. In our example, we compare EBITDA, or earnings before interest, taxes, depreciation and amortisation. The reviewer should be able to trace a reported value back to the company’s report and check how it was calculated. This also allows them to confirm that the figure covers the correct period and scope of operations. 3. Aligning definitions before comparing margins Companies may report adjusted EBITDA that excludes selected costs. These adjustments increase the value of the metric. Before comparing profitability, it is therefore necessary to check which adjustments have been applied. The US SEC also highlights differences in how individual companies calculate financial measures. Let’s look at two fictional companies. We assume that both calculate EBITDA before adjustments using the same principles. Company A then adds back EUR 4.59 million in costs that it excludes when calculating adjusted EBITDA. As a result, the metric rises from EUR 27.57 million to EUR 32.16 million. Company B has no such costs, so its figure remains unchanged. Illustrative example. Consolidated data for the same calendar year. Amounts are stated in EUR million. Item Company A Company B Revenue 229.73 183.78 EBITDA before adjustment 27.57 23.89 Costs excluded when calculating adjusted EBITDA 4.59 0.00 Adjusted EBITDA 32.16 23.89 EBITDA margin before adjustment 12% 13% Adjusted EBITDA margin 14% 13% After the adjustment, Company A’s margin is 14%, exceeding Company B’s margin of 13%. Before the adjustment, Company B has the higher margin: 13% compared with 12%. In this example, the treatment of costs determines which company has the higher EBITDA margin. The analyst should therefore check which costs make up the EUR 4.59 million adjustment and whether they also occurred in previous years. This helps them assess whether excluding these costs is justified for the analysis being prepared. They can also present both scenarios and explain the difference to the client. AI can help gather data and recalculate margins, while the expert assesses whether the adjustment is justified and how it affects the conclusions. 4. Verifying calculations in the spreadsheet In our example, simply divide EBITDA by revenue: 27.57 ÷ 229.73 gives a margin of approximately 12%, while 32.16 ÷ 229.73 gives approximately 14%. The displayed amounts are rounded; the spreadsheet should retain full precision for its calculations. More complex analyses may require currency conversion, alignment of reporting periods or the preparation of several forecast scenarios. The reviewer should be able to trace each of these steps. It is therefore worth asking AI to create a spreadsheet in which source data, assumptions and formulas are clearly separated. The analyst can then check the calculations and see how changing a single value affects the result. To test the spreadsheet, you can halve the adjustment, reducing it from approximately EUR 4.59 million to EUR 2.30 million. Company A’s adjusted EBITDA should then be approximately EUR 29.86 million, with a corresponding margin of 13%. These amounts are rounded for presentation; the spreadsheet should calculate the change using unrounded values. After making this change, check the comparison table and the commentary on the results as well. Both companies would now have the same margin, so the conclusion that Company A has a higher margin would need updating. This is a simple way to assess whether the calculations and accompanying text remain consistent. 5. Preparing client materials and reviewing conclusions The completed analysis can be presented in the company’s preferred format. According to OpenAI’s description, an administrator can share Excel, Word and PowerPoint templates with the team for the tool to use when creating documents and presentations. In our example, the client should receive a results table and a short explanation of how the cost adjustment affects the margin comparison. The expert reviewing the material checks whether the conclusions match the calculations and answer the client’s question. If anything needs clarification, they can request a further explanation or another version of the analysis. Time measurements should also include reviewing the material and making corrections before approval. How can you protect data and preserve a record of the analysis? When preparing a client analysis, the team may use public reports, paid databases and confidential documents. It is necessary to establish who can access this information, where it will be stored and who can receive the finished material. According to the ChatGPT Work security documentation, business data is encrypted and is not used to train models by default. Data retention periods, processing locations and the scope of recorded activity depend on the settings and connected services. Before implementation, check which user and tool actions are logged and which records can be exported. During the pilot, keep the source documents, successive versions of the spreadsheet and the final material, together with a record of who approved it and when. Then check whether this documentation allows you to reproduce the calculations and trace the approval of the analysis. Separately, verify whether system logs allow user and tool activity to be traced to the extent required by the company. How can you assess whether implementation is worthwhile? Start by choosing a task the team performs regularly, such as updating a company comparison after quarterly results are published. Before testing, measure how long this analysis takes using the existing method and define its quality requirements. These findings will provide a baseline for comparison with AI-assisted work. The time needed to review and correct the analysis must be added to the preparation time. OpenAI highlights this in its guidance on assessing the business value of AI, also recommending that implementation and ongoing usage costs be included. In practice, it is worth comparing: Metric What does it tell us? Time from starting the task to approving the analysis Does the client receive the finished material sooner? Include waiting time between stages. Total time spent by the analyst and reviewer Does the team spend fewer hours on preparation, review and corrections? Number of errors affecting the results or conclusions Does the analysis meet the same quality requirements as the existing approach? Accuracy and completeness of source references Can the origins of key figures and information be verified? Time needed to update the analysis How efficiently can new data be incorporated and the calculations and conclusions that depend on it be updated? Cost per approved analysis What is the cost of the finished material, including team time, the tool, data and the share of implementation and maintenance costs allocated to that analysis? The test should cover several tasks of varying difficulty. Define the assessment criteria before it begins. Someone performing the same analysis for a second time already knows the data and some of the answers, which may shorten the time needed. It is therefore worth using comparable tasks and varying the order in which participants work with AI and with the existing method. If AI saves time, check how the team used it. They may have prepared more analyses, responded to clients sooner or reduced overtime. The implementation assessment should show separately how the time saved was used and whether company spending decreased, and by how much. When is it worth starting a pilot? Consider a pilot if the team regularly gathers data from multiple sources and updates similar analyses. Choose a task that takes analysts a significant amount of time, such as comparing data from company reports. Assign a person to lead the pilot and experts to review the results. If the team only occasionally analyses a few annual reports, check whether tools already approved for use within the company are sufficient. Where data retrieval and calculations are already automated, identify a specific task that the new tool could improve. OpenAI makes the product available to financial institutions that meet its access requirements and directs interested companies to its sales team. Pricing, detailed terms and availability for a particular institution in Poland must be confirmed with the provider. Availability information. The implementation decision should be based on the pilot results: the quality of the analyses, the time needed to prepare and review them, and the total cost of the work. The test will also show whether the tool provides access to the data the team needs. Want to explore where AI could improve analysts’ work in your organisation? Talk to the TTMS team about choosing a task for a pilot, connecting the necessary data sources and assessing the results. How does ChatGPT for Financial Services differ from analysing reports in ChatGPT? ChatGPT for Financial Services provides access to selected professional financial data directly within the tool. It also supports references to specific tables and document passages, as well as the preparation of materials using company templates. When assessing its suitability for a team, check whether the available sources cover the companies, periods and metrics the team needs. Does ChatGPT for Financial Services require separate financial data subscriptions? Selected datasets are included in the product. These cover some of the information supplied by the providers named by OpenAI. The company is also developing integrations intended to let institutions use data covered by their existing subscriptions. Before purchasing, confirm which data is included in the offering and which requires additional access rights. Can ChatGPT for Financial Services be used to analyse companies listed on the Warsaw Stock Exchange? This depends on the availability of data for individual companies. Check whether the tool provides their financial statements, relevant metrics and historical data. The launch announcement alone does not confirm full coverage of the Warsaw Stock Exchange. The best way to assess the product’s suitability is to test it on several companies the team regularly analyses. What should you do if data from ChatGPT differs from the figures in a company’s report? Start by comparing the sources, reporting periods, units and definitions of the metrics. A discrepancy may arise, for example, from using standalone rather than consolidated financial data, or from including EBITDA adjustments. Also check whether the company has published an updated report. The analyst should explain the discrepancy and document which value they used and why.
Read