Senior Data Engineer, Observability Engineering

CVS Health•Work At Home-Arizona, AZ

1d•$92,700 - $222,480

About The Position

CVS Health is seeking a highly skilled Senior Data Engineer, Observability Engineering to join the Enterprise Observability Platform organization and help advance the next generation of observability, infrastructure, and security data capabilities. The Senior Data Engineer, Observability Engineering will play a critical role in designing, building, and operating scalable data pipelines and data products that power enterprise observability, operational intelligence, and security analytics across the organization. The Senior Data Engineer, Observability Engineering is a senior individual contributor responsible for developing and optimizing Databricks-based data engineering solutions that ingest, transform, govern, and deliver high-volume telemetry, infrastructure, application, and security data. This role combines deep hands-on technical execution with ownership of engineering excellence, operational reliability, performance optimization, and data platform best practices. Working closely with Observability Engineering, Security Engineering, Infrastructure Engineering, and Data Platform teams, the Senior Data Engineer, Observability Engineering will contribute to the evolution of the enterprise observability lakehouse by building resilient ingestion frameworks, establishing data quality standards, enhancing governance controls, and driving efficient, scalable data processing patterns. The ideal candidate is passionate about engineering high-quality solutions, proactively identifying opportunities for improvement, influencing technical standards, and raising the overall quality, reliability, and maintainability of the platform and team.

Requirements

5+ years of experience building and supporting enterprise data engineering solutions using Python, Spark, PySpark, SQL, and distributed data processing technologies.
5+ years of experience designing, developing, and operating large-scale data pipelines, data ingestion frameworks, and bronze/silver/gold lakehouse architectures across multiple data sources.
4+ years of experience working with Databricks in production environments, including Delta Lake, Unity Catalog, Databricks Asset Bundles, serverless SQL, performance optimization, and platform operations.
3+ years of experience developing high-volume batch and streaming data solutions, implementing data quality controls, troubleshooting pipeline issues, and improving reliability, scalability, and operational efficiency.
2+ years of experience collaborating with engineering teams, participating in code reviews, contributing to technical standards, applying cloud security and access management principles, and communicating technical solutions to both technical and non-technical stakeholders.

Nice To Haves

Experience building Structured Streaming and near real-time ingestion solutions within Databricks or similar cloud-native data platforms.
Experience implementing Databricks governance capabilities, including Unity Catalog, storage credentials, external locations, catalog management, and access control frameworks.
Familiarity with OCSF or similar observability, cybersecurity, or security event normalization frameworks.
Experience implementing CI/CD pipelines for data engineering solutions using Databricks Asset Bundles, GitHub Actions, Azure DevOps, or equivalent automation frameworks.
Prior experience supporting observability, infrastructure, cybersecurity, logging, telemetry, or operational analytics platforms in large-scale enterprise environments.

Responsibilities

Design, build, deploy, and support scalable Databricks-based data pipelines and lakehouse solutions that ingest, transform, and curate observability, infrastructure, and security data from multiple enterprise sources.
Develop and optimize PySpark workloads, Structured Streaming jobs, and bronze/silver/gold data architectures while ensuring high performance, reliability, scalability, data quality, and operational excellence.
Partner with Security, Infrastructure, Observability, and Platform Engineering teams to onboard new data sources, establish governance standards, improve metadata management, and deliver trusted enterprise data products.
Contribute to engineering standards, code reviews, CI/CD practices, monitoring strategies, and platform optimization efforts while proactively identifying and addressing technical debt, process gaps, and operational risks.
Support the ongoing evolution of the enterprise observability lakehouse by promoting best practices for Databricks, Delta Lake, Unity Catalog, data governance, security controls, access management, and cost-efficient data platform operations.