Automated QA Engineer
Your responsibilities:
● RAG Quality Evaluation: Design and curate datasets for RAG evaluation, develop and execute tests for RAG benchmarking, evaluating the accuracy and relevance of AI-generated content and information retrieval.
● Data-Driven Validation: Use data analysis techniques to evaluate test outcomes, track AI performance trends, and detect regression or drift across releases.
● Automation Test Solution Development: Design and implement automated test suites for both OutSystems front-end modules and backend microservices interacting with RaaS from the Galileo platform.
● Regression Testing: Build and maintain automated regression pipelines to ensure continuous platform stability after deployments and LLM-model updates.
● Integration Testing: Verify seamless communication between LUMIN and Galileo APIs, ensuring data integrity and correct processing across layers.
● CI/CD Integration: Embed automated tests into CI/CD pipelines (GitLab CI preferred) to enable consistent, repeatable quality gates before releases.
● Performance & Reliability Testing: Monitor and test system performance and reliability across environments, identifying performance bottlenecks or RAG degradation issues.
We are looking for you, if you have:
● Experience: 4+ years of professional experience in automated software testing, including experience with web, API, and backend systems. Strong understanding of regression and integration testing principles. Proven experience in AI-app testing, specifically with applications using LLMs and RAG.
● Automation Skills: Hands-on experience with automated testing tools (e.g., Playwright, Selenium, or OutSystems test automation frameworks). Experience developing custom test scripts in Python or JavaScript/TypeScript.
● AI Quality Testing: Experience testing LLM and RAG applications, including prompt validation, factuality testing, and output benchmarking.
● Backend & API Testing: Proven experience validating RESTful APIs, microservices, and asynchronous workflows. Familiarity with API testing tools (e.g., Postman, pytest, REST Assured).
● CI/CD & DevOps: Working knowledge of GitLab CI or equivalent systems (e.g., Github Actions), with test automation integrated into build pipelines.
We offer:
● Interesting and challenging projects
● Flexible working hours
● Friendly, non-corporate atmosphere
● Stable working conditions (CoE or B2B)
● Possibility for self-development and promotion in the company
● Rich benefits package
● Possibility to work remotely
We reserve the right to contact the selected candidates.
Why is it
worth joining TTMS?
Flexible working hours
Not everyone is comfortable operating a company during the standard 9 a.m. – 5 p.m. hours. We have been successfully running projects for many years despite diversified work patterns.
Possibility of remote work
At TTMS, desk-to-desk work is what drives us, but we understand if someone prefers a remote or hybrid model. We are equipped with all the necessary tools to perform either at home or in the office.
Great, non-corporate atmosphere
Professionalism, attention to the quality of work and a friendly, informal atmosphere are not mutually exclusive. A flat structure and natural, friendly relationships create the basis for creativity and innovation.
Interesting, international projects
We create solutions for medium and large companies, we are present in Poland, Malaysia, Denmark, the UK, Switzerland, and India – the effects of your work will travel around the world. In TTMS you have an influence on the company’s success.
Together towards climate neutrality
We select subcontractors to minimize the carbon footprint, organize bush plantings, save peat bogs – all to achieve climate neutrality in 2025. We are glad that we are attracting more and more people who also care about sustainable development and environmental protection.
Friendly offices
Our offices are modern spaces that inspire employees and are environmentally friendly. Białystok is the kingdom of Marvel superheroes, Krakow is a soothing tropical forest, Wrocław is an art gallery, and the Lublin branch is a rock “hall of fame”.
Would you like to work for us?
Sending CV
Include information about technical skills, knowledge of programming languages and IT tools, and possibly experience in projects.
Phone Screening
The interview is aimed at assessing the candidate’s suitability for the position and verifying basic information from the CV. It is also an opportunity to discuss the candidate’s expectations and briefly present the job description and working conditions.
HR + technical meeting
We talk about the candidate’s professional experience, but also expectations regarding the new job. This is also the moment to check references and confirm the authenticity of the information provided, which allows you to get to know the candidate better and assess his fit with the company’s organizational culture.
Client company meeting
The candidate is introduced to the client. This stage aims to match the future employee to a specific project, taking into account both their technical skills and design preferences, which maximizes the chances of successful cooperation.
Employment
We sign an agreement that includes the terms of cooperation, scope of duties, remuneration and other key aspects of employment, such as the rights and obligations of both parties. From now on, the candidate becomes an official employee of the company.
Onboarding + first day of work
A comprehensive process of familiarizing a new employee with the organization, team and tools, aimed at making it easier for him to join work as quickly and effectively as possible. In TTMS, we use an internal e-Learning platform to conduct modern onboarding.
Our offices
Warsaw
The heart of our company and our command center
The Management Board of the company and management of individual business areas function here. On a daily basis, it is here that the most important, strategic and business decisions are made, as well as pro-ecological iniciatives. The Warsaw branch – located in Varso Tower – aims to be the first to create a fully eco-friendly office using renewable Energy and environmentally friendly materials, which is confirmed by the BREEM certificate.