Senior Data Engineer

Allied Benefit Systems•Chicago, IL

4d•$155,000 - $170,000•Remote

About The Position

The Senior Data Engineer will design, build, and own the data infrastructure that powers analytics, machine learning, and AI applications across the enterprise. The role will serve as the technical owner of data ingestion, storage, transformation, and reporting infrastructure for the AI team.

Requirements

5+ years of experience in data engineering, software engineering, or related disciplines.
Strong Python development skills.
Advanced SQL expertise.
Experience developing API integrations and data ingestion solutions.
Experience designing and operating production ETL/ELT pipelines.
Experience with cloud-based data platforms (Azure preferred).
Strong understanding of data modeling and warehousing concepts.
Experience with Microsoft Fabric, Azure Data Lake, Power BI, Snowflake, Databricks, or similar technologies.
Experience working with healthcare, claims, provider, or member data.
Familiarity with machine learning workflows and MLOps concepts.
Experience implementing monitoring, observability, and data quality frameworks.
Experience with Git, CI/CD, Infrastructure-as-Code, and modern software delivery practices
Experience with coding assistants, e.g., Claude Code, for feature-oriented development

Responsibilities

Design and implement scalable data pipelines and ETL/ELT processes.
Develop integrations with internal and external data sources, APIs, databases, and enterprise systems.
Build and maintain data ingestion frameworks for structured and unstructured data.
Own data movement and orchestration across platforms such as Azure Storage, Microsoft Fabric, Snowflake, or future enterprise data solutions.
Develop and maintain data models supporting analytics, reporting, and machine learning workloads.
Manage AI-team data storage patterns, including data lake, warehouse, and reporting architectures.
Establish and maintain dataset versioning, lineage, and reproducibility practices.
Create and enforce data quality standards and monitoring procedures.
Design scalable approaches for handling growing data volumes and data sources.
Own the technical infrastructure supporting Power BI and future reporting platforms that the AI team uses for internal reporting or to produce derivative data products for the enterprise.
Create semantic models, curated datasets, and reporting-ready data products.
Partner with analytics stakeholders to ensure high-quality, trusted reporting assets.
Partner with Machine Learning Engineers and Data Scientists to provide high-quality datasets for model development.
Collaborate with business and IT teams to understand evolving data requirements.
Assist in establishing data governance, security, and compliance practices appropriate for healthcare data environments.
Advise business teams during 3rd party data source negotiations.
Other duties as assigned.