QA Engineer (SDET) — AI, Data & Platform Quality

Programmatic QA • Testing for LLMs & Agents • Data Quality • Platform Reliability About Finalytics.ai Finalytics.ai is the leading provider of personalization for the financial industry. Our platform combines data integrations, machine learning, and real-time technology to make digital experiences more relevant and higher-converting for credit unions and banks. We're a growing startup led by industry veterans, building the next generation of AI-driven personalization. Why This Role Is Different QA at Finalytics goes well beyond clicking through a UI. Our platform makes model-driven decisions, runs LLMs and agents that generate content and answer questions, and depends on data pipelines that feed those models every day — and all of it has to be tested programmatically. We're looking for an engineering-minded QA team contributor to help build quality across three areas: our core personalization features, our LLM and agentic capabilities, and the data that powers them. This is a coding role, embedded in the same repo and release flow as our engineers that will report directly to the CTO. You won't just find bugs — you'll build the automated tests, evals, and data checks that let a small team ship trustworthy AI every sprint. Our stack is Python/Django with a JavaScript personalization tag, backed by MySQL, Celery, BigQuery, and AWS. What You'll Do 1. Programmatic QA of Core Features Extend our scenario test runner — a proprietary harness that captures real production personalization requests and replays them across environments, asserting on expected algorithms and content selection. Grow it into automated regression across every client. Write automated tests in Python with pytest across our tiers — unit, integration, HTTP, and end-to-end. Build headless Playwright end-to-end tests to verify how personalized content and tracking render on real client pages. Harden the pre-deploy quality gate and pre-commit checks that block bad changes automatically. 2. Testing & Stand