Senior Data Engineer

Mayo Clinic•Rochester, MN

8d•Remote

About The Position

We are seeking a talented Senior Data Engineer to join our Advanced Data Lake (ADL) team. This is an infrastructure-heavy, hybrid cloud role with Google Cloud Platform (GCP) as a core requirement. You will build and operate enterprise Data Lakehouse platforms that support large-scale analytics and digital transformation. Your responsibilities will include architecting and maintaining automated data pipelines for ingesting, transforming, and integrating complex datasets. You will use DataStream for real-time data movement and Dataflow for processing at scale. Composer/Airflow will be leveraged for seamless scheduling, monitoring, and automation of pipeline operations. Infrastructure provisioning and workflow management will be handled with Terraform and Dataform to ensure reproducibility and adherence to best practices. All code and pipeline assets will be managed through git repositories, with CI/CD automation and streamlined releases enabled by Azure DevOps (ADO). Changes will be governed by ServiceNow processes to ensure traceability, auditability, and operational compliance. Core duties involve working with cross-functional teams to translate business needs into pipeline specifications, building and optimizing data models for advanced analytics, and maintaining data quality and security throughout all processes. You will automate workflow monitoring and proactively resolve data issues, applying strong technical and problem-solving skills. The ideal candidate will have proficiency in Python and SQL, with significant experience in Google Cloud Platform (especially Dataflow and DataStream), Terraform, Dataform, and orchestration with Composer/Airflow. Experience managing code in git repositories, working with Azure DevOps workflows, and following ServiceNow change management processes is required. Strong communication skills and the ability to manage multiple priorities in a remote, team-oriented environment are also necessary. Develops and deploys data pipelines, integrations and transformations to support analytics and machine learning applications and solutions as part of an assigned product team using various open-source programming languages and vended software to meet the desired design functionality for products and programs. The position requires maintaining an understanding of the organization's current solutions, coding languages, tools, and regularly requires the application of independent judgment. May provide consultative services to departments/divisions and leadership committees. Demonstrated experience in designing, building, and installing data systems and how they are applied to the Department of Data & Analytics technology framework is required. Candidate will partner with product owners and Analytics and Machine Learning delivery teams to identify and retrieve data, conduct exploratory analysis, pipeline and transform data to help identify and visualize trends, build and validate analytical models, and translate qualitative and quantitative assessments into actionable insights. This is a full time remote position within the United States. Mayo Clinic will not sponsor or transfer visas for this position including F1 OPT STEM>

Requirements

Proficiency in Python and SQL.
Significant experience in Google Cloud Platform (especially Dataflow and DataStream).
Significant experience with Terraform.
Significant experience with Dataform.
Significant experience with orchestration using Composer/Airflow.
Experience managing code in git repositories.
Experience working with Azure DevOps workflows.
Experience following ServiceNow change management processes.
Strong communication skills.
Ability to manage multiple priorities in a remote, team-oriented environment.
Demonstrated experience in designing, building, and installing data systems and how they are applied to the Department of Data & Analytics technology framework.

Responsibilities

Architecting and maintaining automated data pipelines for ingesting, transforming, and integrating complex datasets.
Using DataStream for real-time data movement and Dataflow for processing at scale.
Leveraging Composer/Airflow for seamless scheduling, monitoring, and automation of pipeline operations.
Handling infrastructure provisioning and workflow management with Terraform and Dataform.
Managing all code and pipeline assets through git repositories.
Enabling CI/CD automation and streamlined releases with Azure DevOps (ADO).
Governing changes through ServiceNow processes.
Working with cross-functional teams to translate business needs into pipeline specifications.
Building and optimizing data models for advanced analytics.
Maintaining data quality and security throughout all processes.
Automating workflow monitoring and proactively resolving data issues.
Applying strong technical and problem-solving skills.
Developing and deploying data pipelines, integrations and transformations to support analytics and machine learning applications and solutions.
Maintaining an understanding of the organization's current solutions, coding languages, and tools.
Providing consultative services to departments/divisions and leadership committees.
Partnering with product owners and Analytics and Machine Learning delivery teams to identify and retrieve data, conduct exploratory analysis, pipeline and transform data to help identify and visualize trends, build and validate analytical models, and translate qualitative and quantitative assessments into actionable insights.