Senior Data Engineer

AIA Contract Documents

28d

About The Position

AIA Contract Documents (ACD) is seeking a Senior Data Engineer to support the buildout of data infrastructure powering upcoming AI initiatives. This role will play a critical part in designing and structuring data systems that enable scalable, high-quality inputs for AI and machine learning models. The ideal candidate brings deep experience in modern data engineering practices, strong familiarity with Databricks, and the ability to operate independently in a fast-evolving environment with limited oversight.

Requirements

• 5+ years of experience in Data Engineering or related field
• Strong expertise in Databricks, including pipeline development, orchestration, and data architecture
• Experience designing data models to support AI/ML applications
• Proficiency in Python (PySpark) and SQL
• Hands-on experience with Azure-based data environments
• Experience with CI/CD tools such as Azure DevOps (ADO)
• Ability to work independently with minimal oversight and ramp quickly in a fast-paced environment

Nice To Haves

• Experience with Databricks Asset Bundles for deployment and environment management
• Familiarity with transitioning CI/CD workflows to GitHub
• Experience working in Agile development environments
• Exposure to AI/ML workflows and data requirements for model development

Responsibilities

• Design and implement scalable data models optimized for AI/ML use cases, ensuring data is structured for effective model training and inference
• Architect and manage data pipelines using Databricks, including orchestration, job scheduling, and workflow optimization
• Develop and maintain robust ETL/ELT processes to support data ingestion, transformation, and delivery across systems
• Leverage Databricks Asset Bundles to manage deployment of data assets (pipelines, jobs, notebooks, and files) across environments
• Collaborate with cross-functional teams to align data architecture with AI initiative requirements and business objectives
• Ensure data quality, integrity, and governance standards are met across all pipelines and datasets
• Contribute to CI/CD practices using Azure DevOps (ADO), with a future transition to GitHub-based workflows
• Participate in Agile development processes, including sprint planning, stand-ups, and iterative delivery
• Other duties as assigned

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed