Data Engineer

Data Engineer (Mid-level) Location: Remote (Quebec/Canada) About Irth Solutions Irth Solutions is a leading provider of cloud-based SaaS software for damage prevention, asset integrity, stakeholder engagement and land management, helping energy, utility, telecom, and infrastructure companies protect their critical network infrastructure. With nearly three decades of industry experience, Irth serves customers across North America and continues to expand its platform with new data-driven and AI-powered capabilities. About the Role We are looking for a Data Engineer to design, build, and maintain data ingestion and processing pipelines in Databricks, transforming high-volume external data sources into clean, structured, reliable inputs for downstream analysis and intelligence. This role will primarily support our Stakeholder Engagement offering, working closely with our Data Scientist and application teams. Key responsibilities 1. Data Pipeline Development (Primary Responsibility) Design, build, and maintain ingestion pipelines from high-volume external APIs, capable of running continuously and reliably at scale. Implement ingestion and transformation workflows using Databricks (Spark/PySpark, SQL, Delta Live Tables), applying medallion architecture patterns (Bronze → Silver → Gold) to move from raw ingested content to clean, structured, analysis-ready data. Build the infrastructure for deduplication and relevance filtering of incoming content, implementing filtering logic and quality criteria defined in collaboration with the Data Scientist. Implement schema evolution handling and data validation rules as data sources and formats change over time. 2. Platform & Storage Implementation Configure and manage Delta Lake storage structures, tables, partitions, and optimization routines (OPTIMIZE, Z-ORDER, VACUUM). Design and evolve data schemas that balance query performance, cost, and maintainability as data volume grows. Maintain clear metadata and documentation of table