Data Engineer
About The Position
The selected candidate will be responsible for assessing feasibility and technical requirements for LINKS → DataLake integration. This includes collaborating with OPH Immunization Program, OPH Bureau of Health Informatics, and STChealth on data specifications and recurring ingestion pipelines. The role involves building and optimizing ETL workflows for LINKS and complementary datasets (e.g., Vital Records, labs, registries), and designing scalable data workflows to enhance data quality, integrity, and identity resolution. The candidate will also implement data governance, observability, and lineage tracking across all pipelines, mentor engineers, support testing activities, and enforce best practices in orchestration and architecture. Documenting and communicating technical solutions to both technical and non-technical stakeholders is also a key responsibility.
Requirements
- 3 years of experience in data engineering and/or data architecture.
- 2 years of experience with Python for ETL and automation (pandas, requests, API integration).
- 2 years of hands-on experience with SQL queries, stored procedures, and performance tuning (Oracle, SQL Server, MySQL preferred).
- 1 year of experience with ETL orchestration tools (e.g., Prefect, Airflow, or equivalent).
- 1 year of experience with cloud platforms (Azure, AWS, or GCP), including data onboarding/migration.
- 1 year of exposure to data lake / medallion architecture (bronze, silver, gold).
- 2 years of experience preparing written documentation and conducting verbal communication for cross-functional collaboration.
Nice To Haves
- 5+ years of data engineering experience.
- Experience integrating or developing REST/JSON or XML APIs.
- Familiarity with CI/CD pipelines (GitHub Actions, Azure DevOps, etc.).
- Exposure to Infrastructure as Code (Terraform, CloudFormation).
- Experience with data governance and metadata tools (Atlan, OpenMetadata, Collibra).
- Experience with public health or healthcare datasets, including PHI/PII handling.
- Familiarity with SAS and R workflows for supporting epidemiologists and analysts.
- Experience with additional SQL platforms (Postgres, Snowflake, Redshift, BigQuery).
- Familiarity with data quality frameworks (Great Expectations, Deequ).
- Experience with real-time/streaming tools (Kafka, Spark Streaming).
- Familiarity with big data frameworks for large-scale transformations (Spark, Hadoop).
- Knowledge of data security and compliance frameworks (HIPAA, SOC 2, etc.).
- Experience working in an Agile/Scrum environment.
Responsibilities
- Assessing feasibility and technical requirements for LINKS → DataLake integration.
- Collaborating with OPH Immunization Program, OPH Bureau of Health Informatics, and STChealth on data specifications and recurring ingestion pipelines.
- Building and optimizing ETL workflows for LINKS and complementary datasets (e.g., Vital Records, labs, registries).
- Designing scalable data workflows to enhance data quality, integrity, and identity resolution.
- Implementing data governance, observability, and lineage tracking across all pipelines.
- Mentoring engineers, supporting testing activities, and enforcing best practices in orchestration and architecture.
- Documenting and communicating technical solutions to both technical and non-technical stakeholders.
Benefits
- Access to a virtual desktop set up (software) will be provided by Lumen’s client, allowing the user access to the required systems and technology.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Education Level
No Education Listed
