Data Engineer

Manulife•Boston, CA

11d•Remote

About The Position

This role focuses on discovering hidden information within massive datasets by leveraging advanced data analytics and machine learning techniques to enable data-driven strategic decision-making. The Data Engineer will be responsible for designing, building, and maintaining data pipelines for ETL/ELT processes, migrating legacy data architectures to cloud platforms like Azure and AWS, and developing/optimizing Spark-based pipelines using Databricks and Synapse. Automation of complex workflows for data ingestion, transformation, and delivery using Airflow, creation of interactive dashboards, and management of various database systems (MSSQL, PostgreSQL, MongoDB) are also key responsibilities. The role involves implementing secure and compliant pipelines with governance policies, integrating data sources via REST APIs, collaborating with data scientists on machine learning pipelines, and working with cross-functional teams to deliver tailored data solutions. Additionally, setting up monitoring and alerting systems, using Docker for application deployment, and identifying/implementing process improvements to reduce costs are expected. This position is eligible for fully remote work from any location in the United States.

Requirements

Master’s degree or foreign equivalent in Computer Science, Electrical Engineering, Electronic Engineering, Information Systems or related field
3 years of experience with Python, SQL, and ETL processes
3 years of experience with cloud platforms including AWS and Azure
3 years of experience using distributed computing frameworks including Spark, Databricks and Synapse
2 years of experience with orchestration tools including Apache Airflow
3 years of experience with data visualization tools including Power BI and Tableau
3 years of experience with database systems including SQL and NoSQL
3 years of experience using data governance, compliance, and security practices
3 years of experience with REST APIs and integration frameworks
3 years of machine learning and predictive modeling experience
3 years of experience with monitoring tools including Splunk and CloudWatch
2 years of experience using DevOps practices and containerization

Responsibilities

Design, build and maintain data pipelines for ETL/ELT processes for large scale datasets
Migrate legacy data architectures to Azure and AWS leveraging cloud-native services for scalability
Develop and optimize Spark based pipelines using databricks for large-scale data processing and Synpse for creating data pipelines
Automate complex workflows for data ingestion, transformation, and delivery using Airflow
Create interactive dashboards to communicate insights to business stakeholders
Design and manage relational MSSQL, PostgreSQL, and NoSQL (MongoDB) databases for diverse use cases
Implement secure and compliant pipelines including governance policies for sensitive data
Integrate various data sources to streamline data processing and analysis using REST APIs
Collaborate with data scientists to build pipelines for training and deploying machine learning models
Work with cross-functional teams to gather requirements and deliver tailored data solutions
Set up monitoring and alerting systems to ensure data pipeline health and reliability
Use Docker to built and deploy applications and ensure consistency across development and production environments
Identify inefficiencies in existing workflows and implement process improvements to reduce costs