Home •Blog
TTMS Blog
TTMS experts about the IT world, the latest technologies and the solutions we implement.
Sort by topics
GPT-6 Cyber: What OpenAI’s Cybersecurity Model Means for Business
OpenAI has introduced GPT-6 Cyber, an AI model designed for cybersecurity work. For companies managing customer portals, payment services, and internal platforms, potential benefits include faster vulnerability investigations, more frequent security reviews, and better support for engineers preparing and testing fixes. Its practical value will depend on whether it helps teams confirm vulnerabilities and test fixes while reducing the time specialists spend on those tasks. What Is GPT-6 Cyber? GPT-6 Cyber is positioned as OpenAI’s specialized model for cybersecurity work. Before its introduction, BusinessToday reported that selected customers were testing an alpha version through Daybreak Red. According to Investing.com, citing Fortune, OpenAI is also developing a separate product to help customers deploy GPT-6 Cyber securely, automate security tasks, and patch vulnerabilities. The product would give OpenAI greater oversight of how its models are used. For businesses, this raises practical questions about how the service would fit into existing security operations: which tasks it could carry out, which actions would require approval, and what information OpenAI would receive about its use. These details would influence both implementation costs and the suitability of the service for handling sensitive company systems. There is already a documented foundation for this approach. OpenAI describes GPT-5.6 Cyber as a model for approved users conducting advanced vulnerability research and security testing. It also offers Codex Security, an application security agent that helps teams investigate vulnerabilities and prepare fixes. These products show how OpenAI already supports security work; GPT-6 Cyber needs its own assessment of capabilities and performance. For a buyer, three parts of an AI security solution deserve separate attention: Part of the solution What it determines What the business should verify Model The analysis and reasoning available for a task Accuracy on relevant security cases, limitations, and cost Access program Who can use particular capabilities and under which conditions Eligibility, approval requirements, and permitted activities Application and integrations The data the system can inspect and the actions it can execute Supported tools, permissions, review controls, and audit records Before purchasing access, check how the solution will connect to the company’s code repositories, monitoring tools, and process for approving changes. Why Cybersecurity AI Matters to Business Leaders Security work competes for engineering capacity. Investigating a suspicious code change, checking a supplier’s advisory, and validating a patch all draw on people who also maintain applications and deliver new features. AI creates an opportunity to move some of the research and preparation into a repeatable workflow, allowing specialists to spend more time on judgment, validation, and decisions about production systems. Faster security work can also help companies keep their services running. A retailer may need to secure a checkout integration without disrupting sales. A manufacturer may need to understand the consequences of updating software connected to operational processes. A financial services company may need clear evidence of what was investigated and how a problem was resolved. In each case, useful automation must fit the business process surrounding the software. OpenAI’s broader investment shows the importance it assigns to this market. On September 3, 2026, the company announced a $1 billion commitment covering subsidized Daybreak access, training, technical support, and partnerships for organizations protecting essential services. Four Business Uses to Evaluate for GPT-6 Cyber The following scenarios are our analysis of where a cybersecurity model could create value. They are proposed evaluation areas whose suitability for GPT-6 Cyber requires testing. Each depends on the capabilities, tools, and permissions available in the eventual deployment. 1. Investigating Vulnerabilities in Business Applications A useful pilot could focus on a small set of applications with a clear business owner. The model would be given approved code and architecture information, then assessed on whether it helps explain suspected weaknesses and identify the evidence needed to confirm them. For example, a team might investigate whether a customer portal consistently enforces access permissions across related services. The business benefit would come from reducing the time needed to reach a defensible conclusion. A helpful result should identify affected components, explain the conditions under which the issue matters, and make it easier for an engineer to reproduce the problem in an authorized test environment. 2. Preparing and Testing Security Fixes Once a vulnerability is confirmed, the next task is to prepare a change that addresses it. A security AI workflow could help draft a patch, identify related code paths, and propose tests for both the security issue and normal application behavior. Existing Codex Security documentation already describes workflows for reviewing changes, validating findings, and preparing fixes, making this a concrete area for evaluating future model improvements. For the business, success means less engineering effort per validated fix while maintaining release quality. Engineers should still review the change and run appropriate tests before it reaches production. 3. Prioritizing Work Using Business Context Using an up-to-date list of systems, their owners, and supporting documentation, an AI system could help connect technical findings with business consequences. This could help the team decide which vulnerabilities to fix first and who should handle them. 4. Helping Teams Investigate Security Incidents Another use to test is building an incident timeline from security alerts and system logs, with internal procedures guiding the investigation. An evaluation could test whether the model links its statements to evidence, distinguishes observations from hypotheses, and identifies missing information. This scenario would require suitable integrations; it should not be assumed to be a built-in GPT-6 Cyber feature. A useful output could help the next analyst continue an investigation or help a service owner understand the affected business process. Decisions such as disabling accounts, blocking traffic, or isolating systems need explicitly assigned authority because they can interrupt legitimate activity. What Could This Look Like in a Customer Portal? Consider a company preparing a new version of a customer portal connected to its order management system. A security review flags a possible weakness in how the portal checks access to order details. In a controlled pilot, an AI system would receive the relevant code, a description of expected permissions, and test accounts containing fictional customer data. The team would assess whether it can help trace the affected logic, produce a clear explanation, and propose a test that demonstrates the problem. If the finding is confirmed, engineers could evaluate its suggested correction and additional regression tests, which check that existing functions still work after a change. The final result should be a reviewed change with evidence: what was wrong, what was modified, which tests passed, and who approved the release. This would require access to the relevant code, a test environment, and a process for assigning confirmed issues to engineers. How to Measure the Business Value of GPT-6 Cyber A pilot should compare the AI-assisted workflow with the team’s current process using comparable cases. Include confirmed vulnerabilities, issues previously dismissed as false alarms, and cases where the evidence is incomplete. Record analyst review time as well as model execution time. An answer produced quickly can still require substantial investigation. The following measures provide a practical basis for a decision: Measure What to record Why it matters Time to assess a suspected vulnerability Elapsed time and analyst effort needed to confirm or dismiss an issue Shows whether the workflow accelerates investigation Finding quality Confirmed findings, false alarms, and missed issues in a reference test set Reveals whether apparent productivity comes with additional errors Time to a tested and approved fix Time from confirmation to a tested, approved correction Connects AI assistance to remediation Engineering effort Hours spent reviewing, correcting, testing, and documenting outputs Makes supervision costs visible Total cost per resolved case Model use, tools, test environments, integration, and staff time Supports a realistic decision about scaling For example, if a pilot saves analyst time but generates extra work for developers, both effects belong in the assessment. If it uncovers more valid vulnerabilities, the organization also needs capacity to fix them. Access, Pricing, and Deployment Questions for Buyers OpenAI’s existing Daybreak documentation requires appropriate organization approval and project access. It distinguishes access to a program from access to a particular model. For GPT-6 Cyber, organizations should verify the applicable requirements directly, including whether access is limited to an evaluation or supports the planned production use. Commercial evaluation should cover pricing, usage limits, supported integrations, and the handling of company data. Teams should establish what source code, logs, and configuration information would be processed; where processing occurs; how long data is retained; and which contractual controls apply. Availability through a particular cloud provider or within an existing subscription should be confirmed for the specific offering. A pilot budget should include implementation and review costs alongside any model usage fees. How to Keep AI Security Work Under Control A business should define the scope of an AI security workflow as carefully as it defines the scope of an external security assessment. Specify the systems it may inspect, the information it may access, and the actions it may take. Separate permission to analyze a problem from permission to change a live service. OpenAI’s published Daybreak guidance recommends isolated environments, monitoring of agent actions, and enforced limits on authorized activity. For an initial business pilot, that supports a controlled test environment, narrowly scoped credentials, review of proposed changes, and records that allow the team to understand what happened. The operating process also needs an owner. Someone must review unresolved findings, decide when additional evidence is required, and stop a workflow that behaves unexpectedly. Training should cover how to challenge an AI-generated conclusion and how to recognize when the system lacks enough information to proceed. Where Businesses Should Start Start with one application and one recurring security task, such as investigating suspected vulnerabilities or testing fixes. Assign someone to review the results and agree how you will measure accuracy, time saved, and the effort required from engineers. A focused pilot can help establish whether the approach is useful enough to expand. Considering AI for your company’s security processes? Talk to TTMS about the task you want to improve, the systems involved, and your data requirements. Together, we can explore a suitable approach and define what a useful pilot should demonstrate. Discuss your AI use case with TTMS How is GPT-6 Cyber different from using a general-purpose AI model for cybersecurity? GPT-6 Cyber is described in published reports as a model focused on cybersecurity, with early testing through OpenAI’s Daybreak Red program. General-purpose models can also assist with security tasks, such as explaining code, reviewing documentation, and analyzing supplied findings. The question for a business is whether the specialist model produces more accurate, useful results on the tasks its team actually performs. Its name alone does not establish that advantage. A meaningful comparison should give both models equivalent information, tools, and time, then have security specialists assess the results. Published evaluations can help inform that comparison, but results from another model should not be attributed to GPT-6 Cyber. Can GPT-6 Cyber replace a cybersecurity team or an external security provider? A company should retain qualified people responsible for assessing risks, validating findings, and approving changes. A model’s analysis depends on the information and tools available to it, which may leave gaps in its understanding of the company’s systems. Security specialists also consider operational priorities, investigate ambiguous evidence, and coordinate the response when something goes wrong. AI assistance may reduce the effort needed for particular tasks, but that needs to be demonstrated in practice. For a business using an external security provider, a useful discussion is how the provider validates AI-generated work and whether any time savings improve the service. Responsibility for protecting systems and resolving problems should remain clearly assigned. Is GPT-6 Cyber useful for a company that mainly uses software from external vendors? Its usefulness would depend on what the company controls and what information it can access. A business using standard cloud applications may have limited visibility into the vendor’s underlying code. Its own responsibilities may include account permissions, configuration, integrations, and custom extensions. Those areas could provide relevant tasks for AI-assisted analysis, subject to the model’s verified capabilities and the available integrations. Before testing, establish which systems the company is authorized to assess and what requires the vendor’s involvement. Findings affecting the vendor’s product should be reported through its security process so that they can be investigated and addressed. Does an AI security review that finds no vulnerabilities mean an application is secure? No. A review can miss vulnerabilities because of incomplete information, limited test coverage, or errors in the analysis. A finding-free report should therefore describe what was examined, which tests were performed, and what remained outside the scope. For example, reviewing selected source files does not establish that the application’s production configuration and access permissions were also checked. The result should be considered alongside other security evidence, including testing and specialist review appropriate to the application’s risk. Record unresolved questions and limitations so that a clean report does not create unjustified confidence. How should a company assess a supplier offering “GPT-6 Cyber-powered” services? Ask the supplier to explain which model it uses, how it accesses that model, and which parts of the service rely on it. Request a demonstration using a representative, authorized task and ask to see the evidence supporting the findings. Establish who reviews the output, who implements corrections, and what happens when the analysis is wrong or incomplete. The service description should also explain how company data is handled and which actions require approval. If the supplier changes the underlying model, ask how it checks that the service still meets the agreed requirements. Evaluate the offer against clear deliverables, such as validated findings and tested fixes, with named people responsible for the work.
Read
AI in Software Test Automation – 2026 Guide
AI in software test automation is moving beyond isolated experiments and becoming part of broader QA and engineering workflows. Teams are using it to support test design, verify scenarios, generate automation code, analyze execution results, and reduce repetitive maintenance work. The value, however, depends on how these capabilities are introduced and governed. This guide explains where AI can support the test automation lifecycle, what organizations should evaluate before adopting a platform, and how to maintain control over test intent, generated code, execution evidence, and ongoing optimization. 1. AI in Software Test Automation: Why It Matters in 2026 As applications and delivery processes become more complex, test automation must cover more than the execution of predefined scripts. Teams also need to keep tests aligned with changing requirements, maintain automation assets, interpret execution results, and decide which scenarios require attention. AI can support these activities by connecting requirements, test design, execution evidence, code generation, and maintenance signals. The value comes from reducing repetitive work while preserving clear ownership, review, and traceability across the test automation lifecycle. For organizations adopting AI in software test automation, the priority in 2026 should be building a controlled workflow rather than adding isolated AI features. Generated outputs need reliable project context, execution-based verification, defined approval gates, and integration with existing repositories and CI/CD processes. 2. What Is AI in Software Test Automation? AI in software test automation refers to the use of capabilities such as natural language processing, contextual analysis, pattern recognition, and AI agents to support the creation, execution, verification, and maintenance of automated tests. Depending on the platform, AI can support different stages of the test automation lifecycle, from interpreting requirements to generating and maintaining automated tests. 2.1 How AI-Powered Testing Differs from Traditional Test Automation Traditional test automation executes predefined scripts according to explicitly programmed steps, conditions, and assertions. This provides predictability and technical control, but creating and maintaining the scripts can require significant engineering effort as requirements, interfaces, test data, and application behavior change. AI-powered testing adds a contextual layer to this process. It can interpret requirements, use project knowledge to prepare test scenarios, support execution-based verification, generate code, and analyze feedback from test runs. Depending on the implementation, AI may also identify unstable tests, suggest updates, or highlight gaps in coverage. The difference is not that AI removes scripts or human involvement. AI-assisted workflows can reduce the manual effort between a requirement and working automation, while QA remains responsible for test intent and engineering retains control over the code entering the repository. 3. Where AI Supports the Test Automation Lifecycle AI in test automation is not a single capability. Understanding its role at each stage of the testing lifecycle helps teams evaluate platforms more accurately and set realistic expectations for adoption. Understanding these capabilities separately helps teams evaluate platforms more accurately and decide where AI can provide useful support without removing human review and technical control. 3.1 Requirements and Test Design AI can translate requirements, acceptance criteria, tickets, and existing QA knowledge into structured test scenarios. This reduces the manual effort involved in turning business requirements into detailed test cases and can help teams identify missing preconditions, expected results, or alternative paths. The quality of the output depends on the available project context. Generated test cases should therefore remain reviewable and should clearly distinguish confirmed information from gaps that require input from QA or business stakeholders. 3.2 Execution-Based Verification A generated test scenario may appear correct while still being impossible to execute against the actual application. Execution-based verification allows an AI agent to replay the proposed steps, observe application behavior, identify relevant elements, and confirm whether the expected result can be reached. This creates technical evidence before automation code is generated. QA still needs to confirm that the verified path represents the intended business scenario. 3.3 Automation Code Generation AI can use test intent and execution evidence to generate automation code. The output should follow the project’s existing conventions, use its reusable components, and remain readable for the engineering team. Generated code should be tested and delivered through the organization’s normal version control and review process. This keeps automation visible, reviewable, and maintainable instead of isolating it inside a proprietary environment. 3.4 AI-Assisted Test Maintenance and Optimization AI can analyze execution history, recurring failures, unstable assertions, selector changes, and overlapping coverage to identify tests that may require attention. It can then recommend repairs, consolidation, or removal. Significant updates should remain visible and subject to review. An automatically modified test may continue to pass while no longer validating the original requirement, so maintenance decisions need clear ownership and approval. 3.5 Reporting, Traceability, and Review AI can help organize execution evidence and connect results with the relevant requirement, test case, code change, and review decision. This gives QA and engineering teams a clearer view of what was tested, how the automation was created, and why it changed. For enterprise use, teams should be able to inspect AI inputs, generated outputs, execution results, and reviewer decisions rather than relying on an opaque automation process. 4. Where AI Creates Business Value in Test Automation AI creates business value in test automation by shortening the path from a requirement to reviewable automation. It can reduce the manual work involved in interpreting tickets, drafting test scenarios, translating test intent into code, and analyzing execution results. This allows QA professionals to contribute their domain knowledge more directly to automation, while engineering teams focus on technical review and repository standards instead of building every test from scratch. The result can be faster automation delivery without creating a separate handover for every scenario. Greater traceability also helps teams understand how test intent becomes automation and whether later changes still reflect the original requirement. The value depends on implementation quality. AI-generated outputs still require reliable project context, execution-based verification, clear ownership, and review before they enter the test suite. Organizations should measure success through reduced manual effort, shorter time from requirement to reviewed automation, acceptance of generated code, lower maintenance effort, and more dependable execution results. 5. How to Evaluate and Integrate an AI Test Automation Platform Choosing an AI testing platform requires more than comparing feature lists. The platform should fit the team’s existing requirements workflow, automation frameworks, repositories, CI/CD processes, security policies, and technical skills. Decision-makers should begin with the workflow they want to support. A platform designed only to generate test cases may not solve the wider challenge of turning requirements into executable, maintainable automation. Teams should evaluate how the tool handles test design, execution-based verification, code generation, review, reporting, and ongoing maintenance. 5.1 Key Criteria for Evaluating an AI Test Automation Tool When evaluating an AI test automation tool, consider: Requirement integration: Can the platform use tickets, acceptance criteria, and existing QA knowledge as test context? Execution model: Does it verify proposed scenarios against the application before generating automation? Code ownership: Does the output remain accessible as standard code in the organization’s repository? Review and governance: Can teams inspect AI inputs, generated outputs, execution evidence, and reviewer decisions? Framework compatibility: Does the platform support the automation technologies already used by the organization? Repository alignment: Can generated code follow existing fixtures, helpers, hooks, naming rules, and quality standards? Deployment and data control: Can the platform operate within the organization’s required infrastructure and AI governance model? Maintainability: Does it use execution feedback to identify unstable, outdated, or redundant automation? Team fit: Can QA define and review test intent while engineering retains control over code quality? These criteria help distinguish a complete AI test automation workflow from an isolated generation feature. 5.2 Fitting AI Testing into Your CI/CD Pipeline An AI testing platform should work with the organization’s existing version control and CI/CD processes. Generated automation should enter the repository through the normal pull or merge request workflow, where it can be reviewed, executed, and approved before being merged. Execution results should return to the relevant code change, test case, and requirement source. This creates traceability between test intent, automation code, execution evidence, and release decisions. Integration may require repository access, execution environments, credentials, pipeline configuration, and approval gates. These requirements should be assessed during implementation rather than assuming the platform can be added without changes to the existing delivery architecture. 6. How to Adopt AI Test Automation with the Right Controls A practical AI test automation strategy should begin with one clearly defined workflow rather than an attempt to automate the entire testing process. Choose a repeatable, business-critical user journey, document the current effort and maintenance burden, and define what successful adoption should improve. Start with requirements and test cases that have clear preconditions, actions, and expected results. AI output depends on the quality of this context, so incomplete requirements and inconsistent testing conventions should be addressed before the workflow is scaled. Define ownership and approval gates from the beginning. QA should remain responsible for test intent and business relevance, while automation engineers or developers review generated code, repository alignment, and technical quality. AI-generated automation should not enter the test suite without execution-based verification and the required human approval. Implementation should include the technical and governance work required to connect the platform with repositories, execution environments, CI/CD, models, and reporting processes. Finally, track practical outcomes such as the time from requirement to reviewed automation, the number of generated tests accepted with limited changes, maintenance effort, recurring failures, and user adoption. Expand to additional workflows only after the initial process is stable, reviewable, and understood by the teams responsible for maintaining it. This shift changes how QA, automation, and engineering teams collaborate, a topic explored further in our guide to how AI is changing the work of developers, testers, and analysts. 7. Challenges and Limitations of AI in Test Automation AI in test automation depends heavily on the quality of the requirements, project context, test data, and repository conventions available to the platform. Incomplete or inconsistent inputs can lead to scenarios that are technically executable but do not reflect the intended business behavior. Execution-based verification can confirm that a proposed path works against the application, but it cannot independently determine whether the path represents the correct test intent. QA still needs to review the scenario, expected result, and alignment with the original requirement. Generated automation may also require adjustment to match project-specific fixtures, helpers, authentication methods, data setup, and coding standards. Teams should expect an implementation and review process rather than treating AI test automation as a plug-and-play capability. Security and governance create additional requirements. Organizations also need clear controls over deployment, model usage, data access, generated code, and approval of AI-supported changes, especially in business-critical or regulated workflows. The strongest implementations use AI to reduce repetitive work while keeping responsibility for test intent, code quality, and release decisions clearly assigned. For a broader view of risk-based testing, traceability, and quality governance, see our QA best practices guide. 8. How Qatana Brings AI Test Automation into One Controlled Workflow Qatana brings test design, execution-based verification, code generation, review, and maintenance into one agentic test automation workflow. It starts with requirements from Jira or GitLab, turns them into automation-ready test cases, and verifies the proposed user path through Playwright execution before generating code. The resulting standard Playwright code is delivered through a pull-ready merge request. QA retains control of test intent, while engineering reviews the technical implementation through the organization’s existing version control process. Execution results remain linked to the requirement, test case, code, and review history, providing traceability across the workflow. Qatana also uses execution results, CI/CD signals, and reviewer feedback to support test maintenance and optimization. The platform runs entirely on-premise and supports the organization’s selected LLM, keeping project context, generated code, and execution evidence within the customer’s controlled environment. Book a demo to see how Qatana turns requirements into execution-validated Playwright automation. 9. Frequently Asked Questions About AI in Software Test Automation Can AI fully replace manual testers? No. AI can support test design, execution, code generation, and maintenance, but testers still define business intent, review outputs, and perform exploratory testing that requires human judgment. How does AI improve self-healing test automation? AI can detect changes affecting automated tests and suggest or generate updates based on execution evidence. Significant changes should remain visible and require review to ensure the test still validates the intended behavior. Is AI-driven testing suitable for regulated industries? Yes, provided the platform supports appropriate governance, traceability, access control, and data protection. Organizations should also evaluate deployment options, model control, audit evidence, and internal compliance requirements. What skills do QA teams need to work with AI testing tools? QA teams need strong test design, domain knowledge, and the ability to review AI-generated scenarios and execution evidence. Familiarity with automation workflows, repositories, and AI limitations is also valuable.
Read
SaaS validation in pharma and GxP requirements
A SaaS application can support quality processes, laboratory work, manufacturing, distribution or clinical operations without requiring the customer to maintain the application in its own data centre. The cloud delivery model does not remove the regulated company’s responsibility for the way the system is used. If an application processes GxP data, affects product quality or patient safety, or supports regulated decisions, the organisation must demonstrate that it is fit for its intended use and remains controlled throughout its lifecycle. SaaS validation in pharma differs from a conventional implementation project. The supplier controls part of the platform, releases updates on its own schedule and may rely on subcontractors. The customer configures processes, roles, data, integrations and use. Responsibilities are shared, but regulatory accountability for the GxP process remains with the regulated organisation. This guide explains when a SaaS system requires validation, how to build a risk-based strategy, which supplier evidence can be used and how to maintain the validated state through frequent software releases. It focuses mainly on GMP environments. Many of the principles can also support GLP, GCP, GDP and pharmacovigilance systems after the applicable regulations and company procedures have been identified. 1. What SaaS validation means in a GxP environment Computer system validation provides documented evidence that a system meets approved user requirements, performs according to its intended use and is controlled in proportion to risk. In a SaaS model, the object of validation is not the supplier’s product in isolation. It is the customer’s specific use of the application, including its configuration, processes, data, interfaces, user roles and operating procedures. The same product may have a different validation status in two companies. One may use it only to arrange meetings. Another may use it to approve specifications, retain laboratory results or generate a record used in batch release. The product name and supplier claims do not determine the validation scope. Intended use, data criticality and the effect of each function on the regulated process do. In practical terms, the validation programme should answer four questions: Are the business and regulatory requirements clear and approved? Have risks to patients, products and data integrity been identified and controlled? Have the configuration and critical processes been tested effectively? Can the organisation maintain control after go-live? Validation is not a permanent certificate attached to an application. It is an evidence-based state associated with a defined version, configuration and use, and maintained through change control, incident management, access management and periodic review. 2. When a SaaS system falls within GxP scope The first step should be system and function classification, rather than ordering a full set of test scripts. The organisation needs to determine whether the application creates, modifies, stores, transfers or reports data used in regulated activities. It must also consider whether a failure, configuration error or unauthorised change could affect product quality, patient safety, data reliability or process compliance. SaaS applications that may have GxP relevance include: electronic quality management systems and eQMS platforms; document management, learning and training systems; LIMS, ELN and analytical data management platforms; clinical operations, safety and case management applications; systems for change control, deviations, CAPA and complaints; ERP, planning and warehouse systems when their data affect GxP operations; electronic record and signature platforms used in regulated processes. Not every function in such a system has the same criticality. A document approval workflow may require more rigorous controls than a supporting dashboard. A risk-based approach directs testing and documentation towards the functions that affect protected records and regulated decisions. 2.1 GxP impact assessment The impact assessment should identify the regulated process, business owner, record types, decisions made from the data and possible consequences of failure. A single GxP or non-GxP label may be too broad for a system that supports several processes. Function-level and data-flow analysis produces a more defensible scope. Functions are likely to have higher impact when they: control a critical process step or provide data used to release it; generate, calculate or approve a result relevant to product quality; retain original data, metadata and the audit trail; manage electronic signatures or approvals; transfer data between systems without an independent completeness check. The outcome should identify critical requirements, the evidence required and the types of change that will trigger reassessment or regression testing. 3. Regulatory basis for pharmaceutical SaaS validation The primary European reference is EudraLex Volume 4 and EU GMP Annex 11. Annex 11 applies to computerised systems used in GMP-regulated activities. It requires applications to be validated and IT infrastructure to be qualified. Decisions about validation scope and data-integrity controls should be based on a justified and documented risk assessment. Annex 11 does not create a separate legal class for SaaS. The delivery model does not remove obligations relating to supplier assessment, validation, security, audit trails, business continuity, archiving and periodic evaluation. The current EudraLex listing identifies the January 2011 revision of Annex 11 as the applicable published text. EMA guidance on GMP and data integrity expressly includes cloud-based applications and storage within the data lifecycle. An organisation should understand where data and metadata are generated, processed, transferred, stored, retrieved and disposed of, and which parties can access them. Validation effort should be scaled to risk. ICH Q9(R1) on quality risk management states that the level of effort, formality and documentation should be commensurate with risk. This principle does not allow required controls to be reduced for budgetary reasons. It allows evidence to be tailored to function criticality, system complexity and uncertainty. GAMP 5 is widely used as industry guidance for applying a lifecycle and risk-based approach to GxP computerised systems. The ISPE overview of GAMP 5 describes a framework intended to support systems that are effective, reliable and fit for use. GAMP 5 is not legislation and does not certify a product. Each organisation still needs a validation strategy based on applicable regulations, intended use and its pharmaceutical quality system. Where a system supports activities regulated in the United States, the scope of 21 CFR Part 11 and the relevant predicate rules also needs to be assessed. FDA guidance on electronic records and electronic signatures explains that Part 11 applies to specified electronic records maintained or submitted under FDA requirements. Not every electronic function is automatically in scope, and a claim that software is Part 11 compliant does not replace an assessment of the configured process. 4. Responsibilities of the regulated company and SaaS supplier In an on-premises implementation, the regulated organisation may control the servers, database, update schedule and many administrative activities. In SaaS, some of these activities are performed by the supplier or its subcontractors. This changes the sources of evidence and the oversight mechanisms, but it does not transfer accountability for the regulated process. Responsibilities should be recorded in the contract, quality agreement or another controlled document. A standard subscription agreement will rarely address every issue relevant to GxP operation. Area Typical supplier responsibility Regulated company responsibility Expected evidence Product development Controlled development lifecycle, product testing and defect management Assess supplier processes and determine whether available evidence can be relied on Audit or qualification report, SDLC description, test summaries Infrastructure Hosting, platform maintenance, monitoring and technical backups Assess the service model, retention needs and recovery requirements Architecture, responsibility model, recovery test results Configuration Configuration functions and product documentation Approve the configuration required for the GxP process Configuration specification, review and tests Access Role, authentication and access-management capabilities Design roles, provision access and perform periodic reviews Role matrix, test evidence and review records Releases Release communication, product testing and deployment Impact assessment, regression decision and approval for use Release notes, impact assessment and regression tests Data Technical storage, export and protection mechanisms Classify data and control retention, completeness and use Data map, export tests and archiving procedures Incidents Detect and manage service-side events Assess GxP impact, take quality actions and make business decisions Tickets, root-cause analysis, CAPA and decision records The responsibility matrix should not leave gaps or create duplicated duties without a named owner. Particular attention is needed where the supplier assumes that the customer will test a control while the customer considers it part of the standard service. 5. SaaS validation in pharma step by step 5.1 Define the intended use The intended-use statement explains why the organisation is implementing the system, which processes it will support, who will use it and which decisions will rely on its data. It needs to be specific. Describing an application only as a quality management system does not show whether it manages documents, CAPA, training, audits, signatures or all of these functions. The boundary should identify interfaces, source systems, GxP records and functions that remain out of scope. A clear boundary reduces later disputes about testing and accountability. 5.2 Map the process and data lifecycle A process map shows where data are created, reviewed, approved, reported and archived. For SaaS, it should also cover transfers between services, APIs, reporting layers, backups, exports and supplier administration tools. The analysis needs to include metadata required to reconstruct an event. A PDF export may be insufficient when it omits change history, signatures, timestamps, relationships or record context. 5.3 Qualify the supplier and service Supplier qualification should be proportionate to system criticality and dependence on the external service. ISO 27001 certificates, SOC reports and security-test results may provide useful evidence. They do not automatically demonstrate that the customer’s GxP workflow and configuration are compliant. The assessment may cover: the supplier’s quality system and assignment of responsibilities; the software development lifecycle, testing, code review and defect handling; release, change and release-note management; security, vulnerability management and incident response; business continuity, backups and recovery testing; subcontractor oversight and data-processing locations; access to documentation, audit support and data on contract termination. Depending on risk, the organisation may use a questionnaire, document review, remote audit or on-site audit. One fixed method is unlikely to suit every supplier. 5.4 Agree requirements and acceptance criteria The user requirements specification should describe the required outcome and measurable acceptance criteria. Requirements need to be testable and traceable. In addition to business functions, the URS should address data integrity, security, audit trails, electronic signatures, retention, export, performance, availability and failure handling where relevant. A statement such as the system has an audit trail is too broad. The requirement should define which events are recorded, whether the record contains the user, date, time, old and new values and reason for change, and who can review and export the history. 5.5 Assess functional and data-integrity risks The risk assessment links requirements to potential failures and controls. For each critical function, the team should determine what can go wrong, the possible effect, how the failure may be detected and which control reduces the risk. Critical requirements need appropriately strong evidence from testing or another assessed source. Risk assessment should not be reduced to a mechanical multiplication of scores. Rationale, process knowledge and the quality of input information matter more than a superficially precise number. Risk needs to be reconsidered when the configuration, process, interface or supplier changes. 5.6 Design the configuration and controls Standard SaaS configuration can reduce the need for custom code, but configuration can still determine how a regulated process behaves. Statuses, approval paths, escalation rules, roles, dictionaries, forms and retention rules should be described, reviewed and controlled. Extensions, scripts, automation and integrations increase the customer’s responsibility. Their owners, repositories, deployment method and test process should be defined. Extensive customisation may also reduce the value of standard supplier evidence. 5.7 Validate migration and interfaces Data migration requires evidence of completeness, accuracy and preservation of record meaning. The strategy should define mapping rules, transformations, rejected records, record-count reconciliation and exception approval. Sampling may be suitable when its scope is justified by risk and data structure. Interfaces should be tested in both directions when two-way flow matters to the process. Tests should cover valid data, transmission failures, duplicates, message retries, time synchronisation and behaviour after a partial failure. 5.8 Test critical processes Testing should demonstrate that requirements are met and controls are effective. Supplier evidence may be leveraged when the organisation understands its scope, version, environment and approval method. There is no value in automatically repeating every product test. The customer must still verify its own configuration, data, interfaces, roles and critical workflows. The test package may include: positive tests of critical business flows; negative tests and exception handling; role and segregation-of-duty tests; audit-trail and electronic-signature verification; data migration and transfer-integrity tests; report, calculation and export tests; failure, recovery and business-continuity scenarios; acceptance tests performed by representative users. IQ, OQ and PQ terminology may be used where it forms part of the organisation’s quality system, but the labels should not replace a logical connection between requirements, risks and evidence. For a typical SaaS service, much of the installation or infrastructure evidence may come from the supplier. The configured use and intended purpose still require customer assessment. 5.9 Approve the validation and go-live decision The validation summary report should identify the system version and configuration, completed activities, test results, deviations, residual risks and release conditions. An open deviation does not always prevent go-live, but risk acceptance needs to be explicit, justified and approved by the appropriate roles. Before release, the organisation should confirm that procedures, support, training, access management, backup arrangements, monitoring and incident handling are ready. Technical validation without operational readiness does not provide sustained control. 6. Validation documentation for a SaaS system The documentation package should provide traceability from intended use and requirements to risks, controls, tests and the release decision. Document names differ between organisations. The important point is whether evidence is complete, approved, current and reproducible during an inspection. Document or record Main purpose SaaS-specific consideration GxP impact assessment Determine whether and to what extent the system is controlled Analyse functions and data flows rather than the product name alone Validation plan Define scope, roles, methods, environments and acceptance criteria Identify supplier evidence and customer activities Supplier assessment Justify reliance on the supplier and service Subcontractors, hosting, SDLC, releases and audit rights User requirements specification Record approved user and regulatory requirements Testable criteria for data, audit trails, signatures and export Risk assessment Link functions to risks and controls Configuration criticality, integrations and automated changes Configuration specification Provide a controlled record of system settings Workflows, roles, rules, extensions and retention settings Test protocols and evidence Demonstrate that requirements are met Version, environment, test data, results and deviations Traceability matrix Connect requirements, risks and tests Identify which evidence originated with the supplier Validation summary report Record the conclusion and release decision Residual risk, conditions and outstanding actions Validated-state plan Define post-release control Release assessment, regression testing, review and retirement Supplier documentation should be accepted deliberately. A supplier test report is useful only when its version, scope, environment, criteria and approval status are understood and relevant to the functions used by the customer. 7. Data integrity and ALCOA principles A SaaS system should support data that can be attributed to an individual, read, associated with the time of the activity, retained as original data or a verified copy and considered accurate. The extended ALCOA+ principles also address completeness, consistency, endurance and availability. The data-integrity assessment should cover: unique accounts and accountability for user actions; time synchronisation and the meaning of time zones; protection of original data and associated metadata; change control and audit-trail review capability; completeness of exports, reports and verified copies; data protection during integration and migration; retention, archiving, retrieval and readability for the required period; data recovery after failure and evidence that restoration works. It is not enough to confirm that an audit-trail feature exists. The organisation should determine whether it is always active for critical events, who can change its settings, how changes are displayed and how it will be reviewed. If supplier administrators have privileged access, their activities also require appropriate oversight. 8. SaaS releases and the validated state The main challenge may not be initial validation, but the pace of subsequent change. A supplier may issue many releases each year and activate some features automatically. The regulated company needs a process that quickly separates immaterial changes from changes affecting a GxP process. A release-management process should include: Obtain release notes and the planned deployment date. Identify affected functions, interfaces, data and requirements. Assess impact on configuration, risk and documentation. Decide the regression scope and required actions. Execute and approve tests before GxP use where this is possible. Update controlled documents, training materials and traceability. Approve the release or implement an agreed risk-control measure. The supplier agreement should provide enough notice, access to a test environment and clear information about release impact. Where updates cannot be deferred, the customer’s process needs a short assessment window, prioritised regression scenarios and a response plan for an unacceptable result. The validated state also depends on routine operational controls, including access reviews, incident management, CAPA, interface monitoring, periodic evaluation, training, audit-trail review, recovery testing and controlled system retirement. 9. Common SaaS validation mistakes 9.1 Relying only on supplier certificates An ISO 27001 certificate or SOC report may support assessment of security and organisational controls. It does not confirm that the customer’s configuration, workflow, report and user roles meet GxP requirements. 9.2 Reusing an on-premises CSV model without adaptation Repeating documentation and tests designed for locally installed software can create work without addressing the main SaaS risks. The strategy should leverage assessed supplier evidence and focus customer activity on configuration, data, integrations and use. 9.3 Leaving responsibility boundaries unclear An imprecise agreement makes it harder to obtain evidence, investigate incidents and assess releases. Responsibilities should also cover subcontractors, privileged support access, data copies and service termination. 9.4 Testing only the expected path A process may work correctly with ordinary data and fail when an integration breaks, a duplicate appears, an approval is withdrawn or a connection is lost. Negative tests and exception handling are important for critical workflows. 9.5 Treating validation as a one-off event Evidence quickly becomes outdated when subsequent releases are not assessed. The validated-state plan should be designed before go-live, rather than after the first major update. 9.6 Confusing availability with recoverability A high availability percentage does not prove that the company can recover complete records and resume a regulated process within the required time. Backup, retention, recovery objectives, data export and manual continuity procedures require separate assessment. 10. GxP SaaS validation checklist [ ] We have defined the intended use, system boundary and process owner. [ ] We have assessed the effect of each relevant function on patients, products and data integrity. [ ] We have identified the applicable GxP rules and electronic record types. [ ] We have documented data, metadata, interfaces, exports and archive flows. [ ] We have qualified the supplier and assessed its quality system and SDLC. [ ] We have reviewed infrastructure providers, subcontractors and data-processing locations. [ ] We have documented responsibilities in the contract and quality agreement. [ ] We have approved testable user requirements and acceptance criteria. [ ] We have completed a documented risk assessment of functions, data and interfaces. [ ] We have approved the configuration, roles, workflows, retention rules and extensions. [ ] We have verified data migration and interface completeness. [ ] We have tested critical workflows, exceptions, permissions, audit trails and signatures. [ ] We have linked requirements and risks to evidence in a traceability matrix. [ ] We have closed deviations or formally accepted the residual risk. [ ] We have approved the validation report and operational readiness before go-live. [ ] We have a process for release-note assessment, change control and regression testing. [ ] We perform periodic supplier, system, access and validated-state reviews. [ ] We have tested backup, recovery, business continuity and data export. [ ] We have defined archiving, retention and controlled service-exit arrangements. [ ] Personnel have been trained for their roles and responsibilities. 11. How to select a SaaS validation partner A validation partner needs to understand both quality requirements and the technical architecture of the service. Familiarity with CSV templates alone is insufficient for assessing integrations, identity design, cloud controls, automatic releases and supplier evidence. Before engagement, check whether the partner can: translate intended use into GxP scope and a risk-based strategy; assess a SaaS supplier, product documentation and shared responsibilities; prepare the URS, risk assessment, plan, tests, traceability matrix and report; connect quality requirements with security, architecture and data integration; design a sustainable process for maintaining the validated state; support QA, IT and business owners without replacing their regulatory decisions. The delivery model should reflect organisational maturity. One company may need an independent strategy review, another a complete documentation and testing workstream, and a third ongoing support for regular SaaS releases. 12. Why TTMS SaaS validation in pharma requires cooperation between quality assurance, process owners, IT, security and implementation teams. TTMS combines computer system validation capabilities with experience in software development, integration, testing and maintenance for regulated industries. Support can include GxP impact assessment, supplier qualification, validation strategy and documentation, architecture and configuration review, test preparation, data-migration assurance and maintenance of the validated state. The engagement can be adapted to a new implementation, a system already in operation or a programme covering several cloud applications. A practical TTMS engagement may include: Inventory of systems, functions and data flows within GxP scope. Gap assessment of evidence, controls and responsibility boundaries. A risk-prioritised implementation roadmap. Preparation and execution of the agreed validation activities. Support for releases, regression testing and periodic reviews. The objective is to produce evidence that reflects actual system use and can be maintained after go-live. Decisions on GxP scope, risk acceptance and system release remain part of the customer’s quality system. 13. Discussing SaaS validation with TTMS If your organisation is planning a cloud implementation or needs to bring an existing application under control, start with the intended use, supplier evidence and release lifecycle. This establishes the scope before detailed documentation and testing begin. Contact TTMS to discuss a validation strategy, SaaS supplier assessment or support for maintaining the validated state in a GxP environment. FAQ Does every SaaS system used by a pharmaceutical company require validation? No. The scope depends on the intended use of the system and its impact on GxP activities. Applications used solely for general administration may be outside the validation scope. However, systems that create, process, store, or manage critical GxP records require documented assessment and appropriate validation controls to ensure fitness for purpose. Can a SaaS product be certified as GxP compliant? No. There is no universal certification that automatically makes a SaaS product GxP compliant. While suppliers may offer features that support regulatory requirements and provide relevant documentation, compliance ultimately depends on how the regulated company configures, uses, and controls the system within its own processes and quality framework. Is the supplier’s ISO 27001 certificate sufficient? No. An ISO 27001 certificate demonstrates that the supplier has an established information security management system, but it does not verify the customer’s specific configuration, critical workflows, data integrity controls, or GxP compliance requirements. It should be treated as one component of supplier qualification rather than proof of validation. How should automatic SaaS updates be validated? Organizations should establish a process for reviewing release information, evaluating potential impact, classifying changes, and performing risk-based regression testing when required. Critical business processes and GxP workflows should be verified before use whenever possible, or within a controlled and justified post-deployment validation window. Does 21 CFR Part 11 apply to every SaaS system? No. The applicability of 21 CFR Part 11 depends on whether the system creates, maintains, modifies, archives, retrieves, or transmits electronic records required by FDA regulations and whether electronic signatures are used. Each application should be assessed against its intended use and the relevant predicate rules to determine Part 11 requirements.
Read
RAG for Chatbots using CrewAI: Notes from a TTMS Tech Talk
Tech Talk is an internal series of technology sessions for TTMS employees, where we share knowledge and project experience. We demonstrate tried-and-tested tools, discuss challenges we have encountered and explain the solutions that have helped us in our work. Topics include artificial intelligence, data analytics, Salesforce, AEM and project management. Presentations are followed by time for questions, discussion and sharing ideas. During the session “RAG for Chatbots Using CrewAI”, held on 17 September, Jakub Kraśniewski, Senior AI Developer at TTMS, discussed improvements to a chatbot using a client’s documentation. He presented the challenges involved in preparing and retrieving information, the solution implemented and the approach to evaluating answer quality. The project involved a company in the education sector whose customers were preparing for a certification exam. The chatbot was intended to help them find information about registration, exam procedures, grading and appeals, thereby reducing the support team’s workload. It used several hundred pages of publicly available PDF documents, mainly in English. The team needed a way to retrieve relevant information from these materials while meeting a response time requirement of around 4 to 5 seconds. 1. How does RAG help a chatbot use company knowledge? Jakub began the presentation by explaining how RAG (Retrieval-Augmented Generation), a method of generating answers using retrieved source material, works. The system finds information relevant to the user’s question and passes it to a language model as context for the answer. In this project, the retrieved material consisted of passages from documentation describing exam rules and procedures. After extracting text from the documents, the system divides it into smaller chunks. An embedding model (an AI model that represents semantic features of text as numbers) converts these chunks into vectors stored in a database. The user’s question is processed in the same way. Comparing these representations allows the system to retrieve passages that are semantically related to the question. Jakub emphasised that the team is responsible for the quality of the material passed to the model. This involves checking whether the text was extracted correctly, whether the chunking preserved the necessary context and whether retrieval provides information useful for answering the question. The challenges the team encountered in the CrewAI-based solution demonstrated the importance of these steps. 2. What made information retrieval difficult in the CrewAI project? After explaining the basics of RAG, Jakub shared his experience from a project using CrewAI, a framework for building AI agent-based systems. He discussed three problems encountered in the configuration used: overly large text chunks, the absence of an additional relevance assessment and errors in PDF text extraction. 2.1 Overly large document chunks In the configuration Jakub described, text was split into chunks of 4,000 characters. The system retrieved five such chunks for each question, passing up to approximately 20,000 characters of source material to the model. A large chunk can contain information on several different topics, making it harder to match it to a specific question. The model generating the answer must then select the relevant information from the supplied content. In this project, the chunking approach therefore needed to be adapted to the structure of the documents and users’ questions. 2.2 No additional assessment of search result relevance Jakub pointed out that the configuration lacked reranking, which involves reassessing and reordering search results according to their usefulness for answering the user’s question. The system can first retrieve a larger number of passages, then assess them further to select those most useful for preparing an answer. Jakub presented this method as a potential improvement whose value should be evaluated by checking both answer quality and response time. 2.3 Incorrect text reading order in multi-column PDFs Another problem involved document text extraction. The tool read multi-column PDFs row by row, merging content from adjacent columns. This disrupted the order of sentences and made subsequent information retrieval more difficult. The resulting text was then split into chunks. The error therefore originated during data preparation and affected the subsequent stages of document processing. This example showed why evaluating RAG quality should begin with comparing the extracted text against the source document. 3. How does response time affect the choice between Classic RAG, Agentic RAG and Graph RAG? A key project requirement was a response time of around 4 to 5 seconds. Jakub discussed three RAG approaches in terms of data preparation costs, the ability to evaluate their operation and the time needed to handle a question. Approach How it works, as discussed during the session What to consider when choosing Classic RAG Retrieves passages from a knowledge base, optionally reranks them and passes the context to the model. Document chunking quality, retrieval relevance and the amount of context provided. Agentic RAG An agent selects tools and a retrieval method, running additional queries as needed. The ability to adapt retrieval to the question, along with the time and cost of additional operations. Graph RAG Retrieval uses a knowledge graph describing entities found in the source material and the relationships between them. The effort required to build and maintain the graph, and how useful the relationships are for answering users’ questions. In an agentic approach, the model can use several tools, such as vector search, keyword search or filtering by metadata describing the document. Additional steps allow the system to expand its search for information, while their number and sequence affect response time. In the graph-based approach presented, some of the work takes place when building the knowledge base. Entities and the relationships between them are extracted from the text. This mechanism also underpins GraphRAG as described by Microsoft. Jakub highlighted the costs of this preparation and the difficulty of manually analysing a complex graph. In this project, the response time requirement favoured further development of classic RAG. The team focused on document chunking and context selection. 4. How does hierarchical document chunking help preserve context? The solution organised the material into three connected levels: pages, paragraphs and sentences. The system retained information about which paragraph each sentence belonged to and which page contained that paragraph. Content was represented in the vector database at different levels of detail. This allowed retrieval to identify both individual sentences and larger passages containing the required information. According to Jakub, the additional cost of storing and processing these representations was acceptable given the volume of material in the project. Finding a relevant sentence made it possible to retrieve its entire paragraph and provide the model with broader context. The system could also retrieve the whole page when needed. Suppose a user asks about the deadline for appealing an exam result. The system finds a sentence specifying the deadline, then retrieves the entire paragraph explaining when the appeal period begins and how to submit an appeal. This allows the model to account for these conditions in its answer. 5. How can you evaluate RAG quality using your own data? In the final part of the presentation, Jakub emphasised the importance of a benchmark, a set of tests used to compare different versions of a solution. He discussed checking retrieval results against information labelled by a human and using a language model to evaluate answers. In practice, it is useful to assess two stages separately. The first concerns retrieval: did the system return a passage containing the required information? The second concerns the answer: did the model use the supplied material correctly? This distinction helps identify which stage needs improvement. In additional information shared after the session, Jakub clarified the testing method and results. The test set included questions covering the full scope of the documentation, along with real user questions collected anonymously during a prototype launch at the beginning of the year. Answer accuracy increased from around 70% to around 98%, an improvement of approximately 28 percentage points. This result applies to the internal test conducted in this project. According to Jakub, the solution also maintained a fast response time. When he shared these details, the chatbot had completed internal testing, and the company planned to make it available to a subset of customers. Reducing the support team’s workload and making information easier to access remained deployment goals. Assessing whether those goals have been achieved requires data from actual use. The embedding model is another component to evaluate. Jakub noted that its selection should take into account the language of the source material and its performance on the team’s own dataset. The choice of this model affects which passages the system retrieves before it begins generating an answer. 6. What can companies implementing a chatbot learn from this experience? The project shows how specific requirements guide RAG development. The expected response time helped narrow down the choice of solution, document analysis revealed problems with text extraction and chunking, and an internal test made it possible to assess the impact of the changes. When planning a similar implementation, it is worth addressing five areas: Source material: check whether document text extraction preserves meaning and reading order. Document chunking: adapt chunk size and the connections between chunks to the structure of the material. User questions: prepare a test set that reflects the tasks the chatbot is intended to support. Response time: establish expectations and account for them when comparing approaches. Quality assessment: check both the relevance of the retrieved information and how it is used in the answer. Let’s talk about AI in your company TTMS is home to experts who, like Jakub, combine technical knowledge with experience from client projects. During Tech Talks, they share solutions tested in practice and apply what they have learned to subsequent implementations. Are you planning a chatbot that uses your company’s documentation, or looking to improve the answer quality of an existing tool? Let’s talk. We will review your materials, users’ needs and business goals to recommend an appropriate way to use AI. Contact the TTMS team! What documents can a RAG chatbot use as a knowledge base? A RAG chatbot can use company policies, product manuals, procedures, FAQs and other materials containing information relevant to its users. Sources may include PDFs, Word documents, website content and knowledge base articles, depending on the integrations available. Scanned documents require optical character recognition (OCR) to turn images of text into searchable content. Tables, diagrams and complex layouts may need additional processing to preserve their meaning. Before adding documents, check that they are accurate, current and approved for the intended audience. Clearly structured materials help the system retrieve information and provide useful context for its answers. How do you keep a RAG chatbot’s knowledge base up to date? Keeping a RAG chatbot up to date requires a process for detecting and processing changes in its source materials. Depending on business needs, updates can run on a schedule or be triggered when a document is added, edited or removed. The system then updates the searchable content and its associated representations, such as embeddings. Version information and effective dates help distinguish current guidance from older material. Deleted or superseded documents should also be removed from active search results, and cached answers may need refreshing. Assigning an owner to each content area helps ensure that someone remains responsible for the information the chatbot uses. Can a RAG chatbot provide sources for its answers? Yes, a RAG chatbot can include links, document titles, page numbers or quoted passages alongside its answers. This requires the system to preserve source information when processing documents and connect retrieved passages to the response. Useful citations let users open the relevant material and check the context for themselves. The system should also verify that each citation supports the claim it accompanies. A source link alone provides no guarantee that an answer accurately reflects the document. During testing, teams should check both answer quality and citation accuracy, including whether users can access the referenced material. How can a RAG chatbot respect access permissions for company documents? A RAG chatbot can use the signed-in user’s identity and access rights to determine which documents it may retrieve. Permission checks should happen before restricted content reaches the language model. The same controls need to cover document previews, citations and any cached responses that contain protected information. When access rights change in a source system, those changes must also be reflected in the chatbot’s retrieval process. Teams should test the solution using accounts with different roles, including users with limited access. These checks help confirm that each person receives answers based on information they are authorised to view. What should a RAG chatbot do when it cannot find an answer? When the available documents provide insufficient information, a RAG chatbot should clearly explain that it cannot answer reliably from its sources. It can ask a clarifying question if the request is ambiguous or suggest a related document that may help. For questions requiring further assistance, it can direct the user to the appropriate team or support channel. The system needs explicit rules for handling incomplete, conflicting or missing information. Testing should include questions whose answers are absent from the knowledge base, so the team can assess this behaviour. Reviewing unanswered questions can also reveal gaps in company documentation and priorities for future updates.
Read
You’ve Deployed Moodle. How Do You Import Your First Training Materials?
Having Moodle installed and configured is only half the journey toward launching training in your company. The next step is to bring real content into it: courses from a previous system, ready-made SCORM packages, or a list of employees who need access to the platform. In this article, we walk through this process step by step — and what to do if you don’t have any training materials yet at all. 1. Before You Start: Identify Your Starting Point for Importing into Moodle The import method depends on exactly what you want to bring into Moodle. In practice, companies find themselves in one of three situations: You already have a ready-made course in SCORM format – produced earlier, purchased from an external provider, or exported from another LMS. In this case, you simply need to import it as an activity within a course. You have a full backup of a course from another Moodle instance (an .mbz file) – e.g., from a test environment, a previous deployment, or from a company that previously managed your platform. In this case, you import the entire course along with its structure. You don’t have either of the above yet – instead, you have presentations, PDF procedures, onboarding recordings, or simply knowledge in your employees’ heads that still needs to be turned into a course. This is the most common situation right after deployment and requires a different approach, which we cover in the final section. The instructions below apply to the first two cases. Option names and their locations may vary slightly depending on the Moodle version, but the main stages of the process remain similar. For certainty, it’s also worth checking the current official Moodle documentation. You’ll find links to the relevant materials further in the article. 2. Importing an Entire Course from a Backup File (.mbz) If you have an .mbz file – a full course backup including section structure, activities, and (optionally) files – the process looks as follows: Log in to Moodle with teacher or manager permissions and go to the course you want to upload content into (this can be a newly created, empty course). From the course menu, select More → Course reuse → Restore. Select the .mbz file – you can upload it from your computer or point to a file already located in the course’s file area. Moodle will guide you through several screens: file confirmation, restore settings (whether to merge the content with the existing course or overwrite it entirely), and selection of which elements should be restored. At the review stage, you’ll see a full list of what will be imported. This is your last chance to go back and make corrections. Once confirmed, the restore process runs in the background – depending on the size of the course, this can take anywhere from a few dozen seconds to several minutes. Important note: a full backup usually does not include user data such as quiz results or forum posts, unless this option was deliberately selected when creating the backup. This is a safeguard that protects against accidentally transferring personal data between environments. You can find more information about creating backups and restoring courses in the official Moodle documentation. A detailed description of the process is available in the materials. 3. Importing Individual Elements from Another Course If you don’t need an entire course, but only selected materials – e.g., a single module, a quiz, or a set of files – Moodle offers a separate Import feature, available in the same place as Restore: Go to the destination course and select More → Course reuse → Import. Search for the source course (you must have editing permissions in it). Select which types of elements you want to transfer: activities, blocks, filters. On the next screen, select the specific items – you can choose a single quiz or file instead of an entire section. Confirm the import and wait for the completion message. This method is convenient when you’re building several similar courses (e.g., for different departments) and want to reuse some shared materials instead of creating them from scratch each time. 4. Importing a SCORM Package as a Course Activity A SCORM package is the most popular format for a ready-made e-learning course – regardless of whether it was created by an external provider, an authoring tool, or exported from another system. The import looks different than for a course backup, because SCORM here is a single activity within a course, not an entire course: Go to the course where the training should appear and turn on editing mode. In the chosen section, click Add an activity or resource and select SCORM package from the list. Upload the .zip file containing the SCORM package. Configure the basic settings: grading method (e.g., based on the highest score or the most recent attempt), the number of allowed attempts, and how the content is displayed (in a new window or within the course page). Save and return to the course – you’ll see the new activity, ready to be opened by users. It’s worth testing the package from a test account before making it available to the target group – especially if the SCORM package comes from an external source and you’re not sure about its compatibility with your Moodle version. 5. Importing a User List and Assigning Roles Materials are only one side of the equation – the other is the people who will use them. Instead of adding each employee manually, you can import a list in bulk: Prepare a CSV file with columns such as first name, last name, email address, and username (the exact required format depends on the platform’s configuration). Go to Site administration → Users → Upload users. Upload the CSV file and review the preview – Moodle will show how it interprets each column and whether there are any data errors. Confirm the import. New accounts will be created automatically, and if the file included the appropriate columns, users can be immediately enrolled in the specified courses. Separately assign roles (e.g., teacher, participant, manager) at the course or category level, if the import didn’t already do so. If the company uses SSO login (e.g., via Azure AD or Microsoft 365), part of this process can be automated through account synchronization – however, that’s a separate topic beyond manual import. You can find more information about the features described above in the official Moodle documentation: Import course data describes moving selected activities and resources between courses, SCORM settings explains adding and configuring SCORM packages, and Upload users describes bulk account creation and course enrollment using a CSV file. Information on Microsoft 365 integration and user synchronization is also described in Microsoft 365 integration. 6. What to Do When You Don’t Have Any Materials to Import Yet This is the most common scenario right after deployment: the platform works correctly from a technical standpoint, but there’s nothing to upload to it yet. The company has knowledge – procedures, presentations, recordings – but not in a format that Moodle recognizes as a ready-made course. In this situation, there are two paths: Prepare the content yourself, with the help of the AI4E-learning tool. The team defines the training goal and provides the source materials they have (documents, presentations, recordings), and the tool generates a course structure from them, complete with audio narration and quizzes, exporting a ready SCORM package – in exactly the format that can be imported using the method described above. Commission course production to the TTMS team. If you need full graphic design, animation, or language versions, custom production covers the entire process: from the script, through graphics and interactions, to export and testing on the platform. Both paths lead to the same end result: a package ready to be uploaded to the platform you already have configured. 7. Summary Launching Moodle is an important step, but that’s when the practical part of the whole process really begins: transferring existing courses, organizing users, and populating the platform with content that employees will actually use. The starting point can be different for every organization. Sometimes it’s ready-made SCORM packages and courses from a previous LMS, and sometimes it’s presentations, documents, procedures, and expert knowledge that still needs to be turned into training. That’s why it’s worth looking at Moodle as part of a larger learning ecosystem within the organization. TTMS can support the entire process: from needs analysis, Moodle deployment and configuration, through migration and integrations, to preparing and launching training content. If source materials already exist, AI4E-learning can additionally help turn them into ready-made courses and significantly shorten the path from company knowledge to training available to employees. If you’re currently wondering how to move from a deployed platform to a fully functioning training environment, let’s talk about your starting point. We’ll help you determine the right scope of implementation and the next steps tailored to your organization’s materials, systems, and needs. FAQ Does importing a course into Moodle also transfer user results and history? It depends on how the course is transferred. Importing content from another Moodle course does not include user data, such as forum posts. When restoring a course from a backup, the scope of transferred information depends on the backup’s content and the available permissions. What’s the maximum size of a SCORM file that can be uploaded to Moodle? This depends on the settings of the specific installation – the upload file size limit configured on the server and within Moodle itself. If needed, this limit can be raised at the platform configuration level. Can I import a course from a completely different LMS, not just from Moodle? It depends on the format in which the source system exports data. If it provides a SCORM package, it can be imported as an activity in a Moodle course. Fully transferring the course structure, roles, and data from another LMS is usually a separate migration project that requires the scope to be determined in advance. What should I do if the import fails with an error? There can be many causes, ranging from file size limits and environment configuration to compatibility issues when restoring course backups. In such a case, it’s worth analyzing the specific error message along with the Moodle environment’s configuration. Can an imported SCORM package be edited later directly in Moodle? No – a SCORM package is a closed content package, and Moodle treats it as a ready-made file to be played back, not editable material. To make changes, you need to modify the course in the tool where it was created (e.g., in AI4E-learning or another authoring tool), export it again, and replace the previous version of the activity in Moodle.
Read
ChatGPT for financial services: what does combining GPT with professional data sources offer?
On 10 September 2026, OpenAI announced ChatGPT for Financial Services, a solution combining GPT-6 Astra with professional financial data sources and tools for preparing analyses. The product was developed in collaboration with Morgan Stanley and Evercore. It is designed for financial institutions, with an initial focus on investment banking and equity research. It allows analyst teams to find data, perform calculations and prepare client materials in one place. This could reduce the time spent gathering information and transferring it between tools. In this article, you will learn: what data and features ChatGPT for Financial Services offers, what preparing a company analysis with GPT could look like, why metric calculations and data sources need to be checked, which stages require an analyst’s review, how to assess whether implementation is worthwhile for your company. How does ChatGPT for Financial Services support analysts? Materials published by OpenAI and its data providers describe several specific use cases: Comparing companies. Daloopa, a provider of company financial data, makes selected data and metrics available for comparing business performance. Source references help analysts verify where the figures come from. Finding companies that meet specific criteria. Daloopa also describes searching for companies by business activity or geographical region. The resulting list can provide a starting point for further market analysis. Preparing client materials. OpenAI describes creating valuation models, research notes and presentations using company templates for Excel, Word and PowerPoint. ChatGPT for Financial Services provides access to selected data from Daloopa, PitchBook, LSEG News and Crunchbase. OpenAI is also developing integrations that will allow institutions to use data covered by their existing subscriptions. These include S&P Capital IQ, LSEG, MSCI, Dow Jones Factiva and Moody’s. From data to company analysis: five steps in the workflow Let’s walk through preparing a comparison of two industrial companies ahead of a client meeting. The analyst needs to assess profitability, explain material differences and prepare a short note with a results table. Using fictional data, we will show what to check during a pilot, from selecting information to approving the final material. 1. Defining the question and scope of the comparison First, we establish which period to compare: the last full year, a six-month period or the trailing twelve months. We also check whether the figures cover the entire corporate group or an individual company, which currency they use and how each metric was calculated. In our example, we use consolidated data for both groups for the same calendar year. Amounts are stated in millions of euros. Before comparing results, we need to check the start and end dates of the reporting periods. One company’s financial year may end in December, while another’s ends in March. The documentation for the US SEC’s EDGAR database also highlights these differences. The analyst must then decide how to account for the mismatch and whether additional data is needed. The agreed approach should be recorded in the instructions for the AI and included with the completed analysis. This gives the model clear guidance and helps the reviewer understand which data was compared and why. 2. Gathering data and identifying its sources For each important figure, record the company and period it relates to, the units used and how it was calculated. A reference to the specific table or explanatory note in the report is also needed. Keeping the source document and its retrieval date makes it easier to review or update the analysis later. According to OpenAI’s description, ChatGPT for Financial Services lets users locate specific tables and document passages, highlighting the information used in the analysis. In our example, we compare EBITDA, or earnings before interest, taxes, depreciation and amortisation. The reviewer should be able to trace a reported value back to the company’s report and check how it was calculated. This also allows them to confirm that the figure covers the correct period and scope of operations. 3. Aligning definitions before comparing margins Companies may report adjusted EBITDA that excludes selected costs. These adjustments increase the value of the metric. Before comparing profitability, it is therefore necessary to check which adjustments have been applied. The US SEC also highlights differences in how individual companies calculate financial measures. Let’s look at two fictional companies. We assume that both calculate EBITDA before adjustments using the same principles. Company A then adds back EUR 4.59 million in costs that it excludes when calculating adjusted EBITDA. As a result, the metric rises from EUR 27.57 million to EUR 32.16 million. Company B has no such costs, so its figure remains unchanged. Illustrative example. Consolidated data for the same calendar year. Amounts are stated in EUR million. Item Company A Company B Revenue 229.73 183.78 EBITDA before adjustment 27.57 23.89 Costs excluded when calculating adjusted EBITDA 4.59 0.00 Adjusted EBITDA 32.16 23.89 EBITDA margin before adjustment 12% 13% Adjusted EBITDA margin 14% 13% After the adjustment, Company A’s margin is 14%, exceeding Company B’s margin of 13%. Before the adjustment, Company B has the higher margin: 13% compared with 12%. In this example, the treatment of costs determines which company has the higher EBITDA margin. The analyst should therefore check which costs make up the EUR 4.59 million adjustment and whether they also occurred in previous years. This helps them assess whether excluding these costs is justified for the analysis being prepared. They can also present both scenarios and explain the difference to the client. AI can help gather data and recalculate margins, while the expert assesses whether the adjustment is justified and how it affects the conclusions. 4. Verifying calculations in the spreadsheet In our example, simply divide EBITDA by revenue: 27.57 ÷ 229.73 gives a margin of approximately 12%, while 32.16 ÷ 229.73 gives approximately 14%. The displayed amounts are rounded; the spreadsheet should retain full precision for its calculations. More complex analyses may require currency conversion, alignment of reporting periods or the preparation of several forecast scenarios. The reviewer should be able to trace each of these steps. It is therefore worth asking AI to create a spreadsheet in which source data, assumptions and formulas are clearly separated. The analyst can then check the calculations and see how changing a single value affects the result. To test the spreadsheet, you can halve the adjustment, reducing it from approximately EUR 4.59 million to EUR 2.30 million. Company A’s adjusted EBITDA should then be approximately EUR 29.86 million, with a corresponding margin of 13%. These amounts are rounded for presentation; the spreadsheet should calculate the change using unrounded values. After making this change, check the comparison table and the commentary on the results as well. Both companies would now have the same margin, so the conclusion that Company A has a higher margin would need updating. This is a simple way to assess whether the calculations and accompanying text remain consistent. 5. Preparing client materials and reviewing conclusions The completed analysis can be presented in the company’s preferred format. According to OpenAI’s description, an administrator can share Excel, Word and PowerPoint templates with the team for the tool to use when creating documents and presentations. In our example, the client should receive a results table and a short explanation of how the cost adjustment affects the margin comparison. The expert reviewing the material checks whether the conclusions match the calculations and answer the client’s question. If anything needs clarification, they can request a further explanation or another version of the analysis. Time measurements should also include reviewing the material and making corrections before approval. How can you protect data and preserve a record of the analysis? When preparing a client analysis, the team may use public reports, paid databases and confidential documents. It is necessary to establish who can access this information, where it will be stored and who can receive the finished material. According to the ChatGPT Work security documentation, business data is encrypted and is not used to train models by default. Data retention periods, processing locations and the scope of recorded activity depend on the settings and connected services. Before implementation, check which user and tool actions are logged and which records can be exported. During the pilot, keep the source documents, successive versions of the spreadsheet and the final material, together with a record of who approved it and when. Then check whether this documentation allows you to reproduce the calculations and trace the approval of the analysis. Separately, verify whether system logs allow user and tool activity to be traced to the extent required by the company. How can you assess whether implementation is worthwhile? Start by choosing a task the team performs regularly, such as updating a company comparison after quarterly results are published. Before testing, measure how long this analysis takes using the existing method and define its quality requirements. These findings will provide a baseline for comparison with AI-assisted work. The time needed to review and correct the analysis must be added to the preparation time. OpenAI highlights this in its guidance on assessing the business value of AI, also recommending that implementation and ongoing usage costs be included. In practice, it is worth comparing: Metric What does it tell us? Time from starting the task to approving the analysis Does the client receive the finished material sooner? Include waiting time between stages. Total time spent by the analyst and reviewer Does the team spend fewer hours on preparation, review and corrections? Number of errors affecting the results or conclusions Does the analysis meet the same quality requirements as the existing approach? Accuracy and completeness of source references Can the origins of key figures and information be verified? Time needed to update the analysis How efficiently can new data be incorporated and the calculations and conclusions that depend on it be updated? Cost per approved analysis What is the cost of the finished material, including team time, the tool, data and the share of implementation and maintenance costs allocated to that analysis? The test should cover several tasks of varying difficulty. Define the assessment criteria before it begins. Someone performing the same analysis for a second time already knows the data and some of the answers, which may shorten the time needed. It is therefore worth using comparable tasks and varying the order in which participants work with AI and with the existing method. If AI saves time, check how the team used it. They may have prepared more analyses, responded to clients sooner or reduced overtime. The implementation assessment should show separately how the time saved was used and whether company spending decreased, and by how much. When is it worth starting a pilot? Consider a pilot if the team regularly gathers data from multiple sources and updates similar analyses. Choose a task that takes analysts a significant amount of time, such as comparing data from company reports. Assign a person to lead the pilot and experts to review the results. If the team only occasionally analyses a few annual reports, check whether tools already approved for use within the company are sufficient. Where data retrieval and calculations are already automated, identify a specific task that the new tool could improve. OpenAI makes the product available to financial institutions that meet its access requirements and directs interested companies to its sales team. Pricing, detailed terms and availability for a particular institution in Poland must be confirmed with the provider. Availability information. The implementation decision should be based on the pilot results: the quality of the analyses, the time needed to prepare and review them, and the total cost of the work. The test will also show whether the tool provides access to the data the team needs. Want to explore where AI could improve analysts’ work in your organisation? Talk to the TTMS team about choosing a task for a pilot, connecting the necessary data sources and assessing the results. How does ChatGPT for Financial Services differ from analysing reports in ChatGPT? ChatGPT for Financial Services provides access to selected professional financial data directly within the tool. It also supports references to specific tables and document passages, as well as the preparation of materials using company templates. When assessing its suitability for a team, check whether the available sources cover the companies, periods and metrics the team needs. Does ChatGPT for Financial Services require separate financial data subscriptions? Selected datasets are included in the product. These cover some of the information supplied by the providers named by OpenAI. The company is also developing integrations intended to let institutions use data covered by their existing subscriptions. Before purchasing, confirm which data is included in the offering and which requires additional access rights. Can ChatGPT for Financial Services be used to analyse companies listed on the Warsaw Stock Exchange? This depends on the availability of data for individual companies. Check whether the tool provides their financial statements, relevant metrics and historical data. The launch announcement alone does not confirm full coverage of the Warsaw Stock Exchange. The best way to assess the product’s suitability is to test it on several companies the team regularly analyses. What should you do if data from ChatGPT differs from the figures in a company’s report? Start by comparing the sources, reporting periods, units and definitions of the metrics. A discrepancy may arise, for example, from using standalone rather than consolidated financial data, or from including EBITDA adjustments. Also check whether the company has published an updated report. The analyst should explain the discrepancy and document which value they used and why.
ReadRecommended articles
The world’s largest corporations have trusted us
We hereby declare that Transition Technologies MS provides IT services on time, with high quality and in accordance with the signed agreement. We recommend TTMS as a trustworthy and reliable provider of Salesforce IT services.
TTMS has really helped us thorough the years in the field of configuration and management of protection relays with the use of various technologies. I do confirm, that the services provided by TTMS are implemented in a timely manner, in accordance with the agreement and duly.
Ready to take your business to the next level?
Let’s talk about how TTMS can help.
Monika Radomska
Sales Manager