Test Automation Engineer
Your responsibilities:
- Generative AI Testing Strategy: Design and implement comprehensive testing strategies specifically tailored for LLM-based applications, including evaluation of model responses, RAG pipelines accuracy, and overall system reliability
- Quality Assurance Framework Development: Utilize Client’s testing frameworks that address both traditional software quality aspects and AI-specific concerns such as output consistency, contextual accuracy, and ethical compliance; co-create and maintain such frameworks when required
- Test Automation Development: Design and implement automated testing solutions for continuous evaluation of LLM applications, including integration tests, performance tests, and specialized AI behavior tests
- Edge Case Analysis: Identify and develop test scenarios for edge cases in LLM behavior, including handling of ambiguous inputs, potential biases, and unexpected response patterns
- Bias and Fairness Testing: Design and execute tests to identify potential biases in model outputs and ensure fair treatment across different user groups and use cases
- Security Testing: Collaborate within development teams to test for potential vulnerabilities specific to LLM applications, including prompt injection, data leakage, and other AI-specific security concerns
- Test Documentation: Create and maintain comprehensive test documentation testing strategy, test cases, and testing guidelines specific to AI applications and compliant with Client practices; document the analysis of requirements and risks and develop tests accordingly
- Performance Testing: Collaborate with other engineers to conduct thorough performance testing of GenAI applications, including response time analysis, load testing, and resource utilization monitoring
We are looking for you, if you have:
-
Experience: 3+ years of experience in software testing, with at least 1 year focused on genAI applications
-
Technical skills: Strong proficiency in Python and testing frameworks (e.g., pytest, unittest); experience with AI testing tools and frameworks
-
AI/ML knowledge: Understanding of LLM architectures, RAG systems, and common failure modes in AI applications; familiarity with evaluation metrics for generative AI
-
Test Automation: Experience with test automation frameworks and CI/CD pipelines; ability to design and implement automated testing solutions for AI applications
-
Ability to analyze requirements, identify test scenarios, and prioritize testing activities based on risk and impact
-
Cloud Platforms: Practical experience with testing applications on cloud platforms (AWS, Azure) and working with cloud-based AI services
-
Monitoring tools: Experience with monitoring and observability tools, log analysis, and performance metrics tracking (e.g. DataDog, Grafana)
-
ALM tools: Experienced in Roche mandatory ALM tools (e.g. Jira, GitLab, ServiceNow)
-
Security Testing: Understanding of security testing methodologies, particularly in the context of AI applications
- Very good practical and theoretical knowledge of testing processes (ISTQB Advanced, etc.)
- The successful candidate should also hold B.Sc., B.Eng., M.Sc., M.Eng. in Computer Science, Software Engineering, or equivalent degree with strong background in testing methodologies
- Be able to communicate in English at the level of C1+
We offer:
- Interesting and challenging projects
- Flexible working hours
- Friendly, non-corporate atmosphere
- Stable working conditions (CoE or B2B)
- Possibility for self-development and promotion in the company
- Rich benefits package
- Possibility to work remotely
We reserve the right to contact the selected candidates.
Why is it
worth joining TTMS?
Flexible working hours
Not everyone is comfortable operating a company during the standard 9 a.m. – 5 p.m. hours. We have been successfully running projects for many years despite diversified work patterns.
Possibility of remote work
At TTMS, desk-to-desk work is what drives us, but we understand if someone prefers a remote or hybrid model. We are equipped with all the necessary tools to perform either at home or in the office.
Great, non-corporate atmosphere
Professionalism, attention to the quality of work and a friendly, informal atmosphere are not mutually exclusive. A flat structure and natural, friendly relationships create the basis for creativity and innovation.
Interesting, international projects
We create solutions for medium and large companies, we are present in Poland, Malaysia, Denmark, the UK, Switzerland, and India – the effects of your work will travel around the world. In TTMS you have an influence on the company’s success.
Together towards climate neutrality
We select subcontractors to minimize the carbon footprint, organize bush plantings, save peat bogs – all to achieve climate neutrality in 2025. We are glad that we are attracting more and more people who also care about sustainable development and environmental protection.
Friendly offices
Our offices are modern spaces that inspire employees and are environmentally friendly. Białystok is the kingdom of Marvel superheroes, Krakow is a soothing tropical forest, Wrocław is an art gallery, and the Lublin branch is a rock “hall of fame”.
Would you like to work for us?
Sending CV
Include information about technical skills, knowledge of programming languages and IT tools, and possibly experience in projects.
Phone Screening
The interview is aimed at assessing the candidate’s suitability for the position and verifying basic information from the CV. It is also an opportunity to discuss the candidate’s expectations and briefly present the job description and working conditions.
HR + technical meeting
We talk about the candidate’s professional experience, but also expectations regarding the new job. This is also the moment to check references and confirm the authenticity of the information provided, which allows you to get to know the candidate better and assess his fit with the company’s organizational culture.
Client company meeting
The candidate is introduced to the client. This stage aims to match the future employee to a specific project, taking into account both their technical skills and design preferences, which maximizes the chances of successful cooperation.
Employment
We sign an agreement that includes the terms of cooperation, scope of duties, remuneration and other key aspects of employment, such as the rights and obligations of both parties. From now on, the candidate becomes an official employee of the company.
Onboarding + first day of work
A comprehensive process of familiarizing a new employee with the organization, team and tools, aimed at making it easier for him to join work as quickly and effectively as possible. In TTMS, we use an internal e-Learning platform to conduct modern onboarding.
Our offices
Warsaw
The heart of our company and our command center
The Management Board of the company and management of individual business areas function here. On a daily basis, it is here that the most important, strategic and business decisions are made, as well as pro-ecological iniciatives. The Warsaw branch – located in Varso Tower – aims to be the first to create a fully eco-friendly office using renewable Energy and environmentally friendly materials, which is confirmed by the BREEM certificate.