• Generative AI Testing Strategy: Design and implement comprehensive testing strategies specifically tailored for LLM-based applications, including evaluation of model responses, RAG pipelines accuracy, and overall system reliability
• Quality Assurance Framework Development: Utilize Client’s testing frameworks that address both traditional software quality aspects and AI-specific concerns such as output consistency, contextual accuracy, and ethical compliance; co-create and maintain such frameworks when required
• Test Automation Development: Design and implement automated testing solutions for continuous evaluation of LLM applications, including integration tests, performance tests, and specialized AI behavior tests
• Edge Case Analysis: Identify and develop test scenarios for edge cases in LLM behavior, including handling of ambiguous inputs, potential biases, and unexpected response patterns
• Bias and Fairness Testing: Design and execute tests to identify potential biases in model outputs and ensure fair treatment across different user groups and use cases
• Security Testing: Collaborate within development teams to test for potential vulnerabilities specific to LLM applications, including prompt injection, data leakage, and other AI-specific security concerns
• Test Documentation: Create and maintain comprehensive test documentation testing strategy, test cases, and testing guidelines specific to AI applications and compliant with Client practices; document the analysis of requirements and risks and develop tests accordingly
• Performance Testing: Collaborate with other engineers to conduct thorough performance testing of GenAI applications, including response time analysis, load testing, and resource utilization monitoring

• Experience: 3+ years of experience in software testing, with at least 1 year focused on genAI applications

• Technical skills: Strong proficiency in Python and testing frameworks (e.g., pytest, unittest); experience with AI testing tools and frameworks

• AI/ML knowledge: Understanding of LLM architectures, RAG systems, and common failure modes in AI applications; familiarity with evaluation metrics for generative AI

• Test Automation: Experience with test automation frameworks and CI/CD pipelines; ability to design and implement automated testing solutions for AI applications

• Ability to analyze requirements, identify test scenarios, and prioritize testing activities based on risk and impact

• Cloud Platforms: Practical experience with testing applications on cloud platforms (AWS, Azure) and working with cloud-based AI services

• Monitoring tools: Experience with monitoring and observability tools, log analysis, and performance metrics tracking (e.g. DataDog, Grafana)

• ALM tools: Experienced in Roche mandatory ALM tools (e.g. Jira, GitLab, ServiceNow)

• Security Testing: Understanding of security testing methodologies, particularly in the context of AI applications

• Very good practical and theoretical knowledge of testing processes (ISTQB Advanced, etc.)
• The successful candidate should also hold B.Sc., B.Eng., M.Sc., M.Eng. in Computer Science, Software Engineering, or equivalent degree with strong background in testing methodologies
• Be able to communicate in English at the level of C1+

• Interesting and challenging projects
• Flexible working hours
• Friendly, non-corporate atmosphere
• Stable working conditions (CoE or B2B)
• Possibility for self-development and promotion in the company
• Rich benefits package
• Possibility to work remotely

We reserve the right to contact the selected candidates.