Senior Data/ML Engineer — Databricks Forecasting Platform | Remote

<img src="https://we-work-remotely.imgix.net/logos/0171/6036/logo.gif?ixlib=rails-4.0.0&w=50&h=50&dpr=2&fit=fill&auto=compress" /> <p> <strong>Headquarters:</strong> Remote <br /><strong>URL:</strong> <a href="https://www.toptal.com/">https://www.toptal.com/</a> </p> <h2>About the Role</h2> <p>We're looking for a Senior Data/ML Engineer to refactor, operationalize, and improve an existing time series forecasting platform that's already live and driving real business value. This is not a greenfield build — the focus is modernizing a Databricks-based forecasting system that's been maintained primarily by a single developer for years. You'll reduce technical debt, strengthen testing and observability, and raise the engineering bar on a system the business already depends on. If you'd rather bring discipline and maturity to an existing production system than start from a blank slate, this is built for that.</p> <h2>What You'll Do</h2> <ul> <li> <p>Review the current forecasting platform architecture and identify areas for improvement</p> </li> <li> <p>Refactor existing Databricks, Python, and PySpark implementations</p> </li> <li> <p>Move business logic out of Databricks notebooks and into reusable Python modules or packages</p> </li> <li> <p>Improve separation of concerns between orchestration and core business logic</p> </li> <li> <p>Establish stronger engineering standards and help define what "good" looks like for the platform</p> </li> <li> <p>Implement or improve automated testing practices and validation mechanisms for forecasting workflows</p> </li> <li> <p>Build or improve monitoring and observability, increasing visibility into how predictions are generated</p> </li> <li> <p>Help monitor model behavior and operational health over time</p> </li> <li> <p>Improve reliability of scheduled training workflows, reducing manual intervention on failure</p> </li> <li> <p>Improve failure handling, retries, and overall workflow resilience</p> </li> <li> <p>Maintain and ex