Sr. Data Engineer
About The Position
We are seeking a highly skilled Senior Data Engineer to join our Data Engineering & Platform organization. This role is ideal for someone passionate about building scalable data solutions, enabling cross-functional data delivery, and raising engineering excellence across the organization. You will work on an advanced Databricks-on-AWS data platform and play a key role in transforming our data ecosystem—designing and building resilient pipelines, implementing CI/CD, and contributing to platform modernization efforts. As a senior member of the team, you will partner with data engineers, analytics teams, product teams, and platform engineering to build high-quality, production-grade data solutions that power enterprise analytics, data science, and operational workloads.
Requirements
- 7+ years of data engineering experience building production-grade data pipelines.
- Deep expertise in Databricks , Spark , and Delta Lake .
- Hands-on experience with AWS cloud services.
- Strong proficiency in Python , SQL , and distributed computing patterns.
- Experience with Terraform and cloud-native CI/CD.
- Strong understanding of data modeling, orchestration, and data quality frameworks.
- Ability to operate autonomously while collaborating in a team environment.
Nice To Haves
- Experience with Lakeflow, Databricks Workflows, or ingestion modernization initiatives.
- Knowledge of software engineering best practices: test automation versioning code reviews Git workflows
- Familiarity with event-driven architecture and streaming technologies.
- Experience supporting Data Science and ML platform teams.
- P&C Insurance domain knowledge preferred
Responsibilities
- Design, build, maintain, and optimize scalable data pipelines and ELT/ETL processes to ingest, process, and store large volumes of data from various sources.
- Architect robust batch and streaming solutions using PySpark, Spark SQL, and Databricks Jobs.
- Monitor and troubleshoot data pipelines and infrastructure to ensure high availability and performance.
- Ensure reliability, observability, and SLAs through monitoring, alerting, and automated recovery patterns.
- Implement and maintain data integration solutions, including APIs and data connectors, to facilitate data exchange between systems.
- Build and manage data solutions on AWS, leveraging services such as S3, Lambda, Glue, IAM, EC2, EKS, Step Functions, and EventBridge.
- Apply cloud engineering best practices for cost optimization, reliability, and security.
- Implement and maintain CI/CD pipelines using: Databricks Asset Bundles Terraform (IaC) GitHub Actions / Azure DevOps / Jenkins (depending on internal stack)
- Own deployment automation and environment management for data workloads and platform components.
- Contribute to reusable Terraform modules and engineering standards.
- Maintain data quality through validation frameworks (e.g., DQX, Great Expectations, Databricks expectations, custom frameworks).
- Ensure data quality, integrity, and security by implementing best practices for data governance and management.
- Implement governance and lineage through Unity Catalog, cataloging standards, and metadata best practices.
- Contribute to initiatives such as: Migration of pipelines from legacy integration tools to Lakeflow / Lakehouse-native ingestion Improving infrastructure automation Enhancing data observability, testing, and deployment workflows
- Serve as a technical mentor to junior engineers.
- Partner closely with data analysts, data scientists, and business stakeholders to deliver enterprise data assets.
- Contribute to architecture decisions, design reviews, and platform roadmap conversations.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Education Level
No Education Listed
