Senior QA Automation Engineer (AI & LLM Testing)

👋🏼 We're Nagarro. We are a digital product engineering company that is scaling in a big way! We build products, services, and experiences that inspire, excite, and delight. We work at scale — across all devices and digital mediums, and our people exist everywhere in the world (18 000+ experts across 39 countries, to be exact). Our work culture is dynamic and non-hierarchical. We're looking for great new colleagues. That's where you come in! By this point in your career, it is not just about the tech you know or how well you can code. It is about what more you want to do with that knowledge. Can you help your teammates proceed in the right direction? Can you tackle the challenges our clients face while always looking to take our solutions one step further to succeed at an even higher level? Yes? You may be ready to join us. We are looking for a Senior QA Automation Engineer with strong hands-on experience in test automation, API testing, and AI/LLM quality evaluation. Design, develop, and maintain automated test suites for web applications, APIs, and AI-powered features. Evaluate LLM outputs for response accuracy, relevance, consistency, and hallucination risks. Create prompt tests and quality checks using Promptfoo or similar LLM evaluation frameworks. Build end-to-end UI automation using Playwright, following page object models and maintainable test architecture principles. Develop reusable Python scripts for API testing, automation utilities, and LLM quality evaluation. Create and maintain Postman collections for REST API testing, including environments, assertions, and CI/CD integration. Use GitHub Copilot or similar AI coding assistants to improve test coverage, speed up automation development, and identify quality gaps. Collaborate closely with product managers, developers, and stakeholders to define test strategies, communicate risks, and improve quality practices. Actively contribute to Agile ceremonies, including refinement sessions, sprint planning, retro