AI ML Data Engineer

Cloudious • Jersey City, NJ

1d

About The Position

The Senior Data Engineer will focus on data engineering and AI/ML solutions, building and maintaining data pipelines, data models, and data integration workflows. The role contributes to modernizing data platforms and integrating AI capabilities into enterprise systems.

The position involves designing and maintaining batch and streaming pipelines; leading migrations from on-premise platforms to cloud-native architectures; developing core data platform components; and working with data domain experts to create connected data models. It also supports data quality, governance, analytics, reporting, and AI/ML use cases, while optimizing production data and AI pipelines for scalability, performance, reliability, and cost efficiency. The engineer will partner with AI/ML Engineers and cross-functional stakeholders, and mentor engineers and interns.

The posting lists Jersey City, New Jersey, and a duration of 6+ months.

Requirements

• The role overview lists 10+ years of experience with Databricks, Azure Data Factory (ADF), and SQL.
• At least 5 years of software development experience focused on Data Engineering.
• At least 1 year of experience delivering solutions that include AI/ML or generative components.
• Proven experience building and maintaining ML pipelines, data modeling, and data integration workflows.
• Basic SQL skills for querying, transformation, and performance optimization.
• Advanced proficiency in Python and experience with Scala for data and AI workloads.
• Hands-on experience with Spark batch and streaming, Databricks, SparkSQL, and cloud platforms.
• Experience with Azure IaaS, cloud infrastructure, monitoring, and deployment automation.
• Experience with CI/CD, version control, and DevOps practices, including Git, GitHub Actions, or Jenkins OpenSource.

Nice To Haves

• Experience with Databricks ML and Agentic AI solutions.
• Knowledge of master data management (MDM), data governance, and data governance frameworks.
• Experience in healthcare, insurance, or other regulated industries.
• Familiarity with Intersystems HealthShare, HL7 reporting standards, and healthcare data standards.
• Experience with Terraform, infrastructure as code, and robotic process automation.
• Azure Databricks certifications.
• Strong stakeholder management and cross-functional collaboration skills.
• Knowledge of data security, privacy, and Enterprise Architecture best practices.
• Proven success delivering master data, analytics, or AI-enabled data products using Azure Databricks, Snowflake, and cloud platforms.
• Experience with Data Governance, MDM, Data Mesh, and regulated environments.

Responsibilities

• Design, build, and maintain scalable, secure, high-performance batch and streaming data pipelines.
• Lead modernization and migration of on-premise data platforms to cloud-native architectures, including infrastructure, networking, security, and deployment pipelines.
• Develop and maintain data platform components for ingestion, transformation, enrichment, and consumption of enterprise data.
• Collaborate with data domain experts to understand data sources and design efficient, connected data models.
• Implement validation, monitoring, documentation, and security controls to support data accuracy, quality, relevance, and governance.
• Analyze large datasets to identify trends, patterns, and actionable insights for analytics and AI use cases.
• Make curated, trusted datasets available to BI and downstream consumers.
• Partner with AI/ML Engineers to design, build, and improve machine learning models and AI-powered enterprise applications.
• Optimize production data and AI pipelines for scalability, performance, reliability, and cost efficiency.
• Mentor and guide engineers and interns in data engineering and AI/ML.
• Collaborate with cross-functional teams and stakeholders, clearly communicating complex technical concepts.

What This Job Offers

• Career Level: Senior
• Education Level: No Education Listed