Applied AI/ML Engineer

YOUR ROLE As a founding AI Engineer at Overtone, you will build and own the AI systems that power our core product experience—from conversational intelligence to matchmaking insight generation. You'll design prompting, evaluation, and model infrastructure that makes our AI reliable, trustworthy, and continuously improving. This is a hands-on, early-stage role: you'll run experiments, ship infrastructure, and build the tools that give the team real visibility into model behavior and quality. What You’ll Do - LLM systems & prompt architecture: Design and maintain the prompts, structured outputs, and orchestration systems that power Overtone's AI-driven features. - Evaluation frameworks: Build evaluation infrastructure to measure AI quality at scale using tools such as Langfuse, TensorZero, and Promptfoo. You'll work with our team to make sure LLM judges are aligned. - Model experimentation: Run structured experiments across models, prompts, and configurations to optimize quality, cost, and latency. - AI observability & tooling: Build internal tools—dashboards, admin interfaces, debugging workflows—that give the team visibility into system behavior and model performance. - Matching & ML systems: Apply data science and lightweight modeling to improve match quality, identifying when traditional ML models or ranking systems outperform prompt-based approaches. - Model serving & production inference: Own the model serving layer — deploying models, managing inference infrastructure (API providers and self-hosted), model versioning, and cost/latency optimization in production. Work with the backend lead to define clean service contracts between the AI layer and the rest of the system. - Model development & fine-tuning: Identify opportunities to fine-tune open-source models using techniques such as SFT, DPO, and RLHF for cost and latency improvements. - Memory & personalization systems: Design and maintain vector databases and memory architectures that enable personalization a