Senior Data Quality Engineer (I, II, III)

Cellares•South San Francisco, CA

61d•$90,000 - $210,000

About The Position

Cellares is seeking an innovative and highly motivated Senior Data Quality Engineer who will contribute to the development of our advanced cell therapy manufacturing platform. The primary focus of this position is to ensure the accuracy, reliability, and integrity of data within our data platform. The individual will participate on a cross-functional team, design, build, and maintain automated testing frameworks to ensure data integrity at every stage of our data pipelines. The successful candidate should have extensive experience in quality assurance for data platforms, ideally with significant hands-on experience in the Databricks environment. They should be detail-oriented and possess strong analytical and problem-solving skills. Candidates should enjoy working in a fast-paced, mission-driven environment, and be prepared to tackle a broad selection of challenges as the company grows. Candidates should be great team players with the ability to work with minimal supervision.

Requirements

Bachelor's or Master's in Computer Science, Electrical Engineering, or related field and 5+ years of relevant experience
Experience with data pipeline and data quality testing strategy and execution, with significant hands-on experience in the Databricks environment
Strong proficiency in Python for developing and executing data validation scripts
In-depth knowledge of Databricks, Delta Lake, and the Lakehouse architecture. Proficiency in writing complex SQL queries for data validation, reconciliation, and troubleshooting issues
Solid understanding of data warehousing concepts, including dimensional modeling (star/snowflake schemas)
Hands-on experience with Azure, including Azure storage and data services that integrate with Databricks
Ability to process data, interpret testing results and provide feedback to the team
Desire to be part of a rapidly evolving organization, with compelling technology, and taking products and processes to the next level
Self-awareness, integrity, authenticity, and a growth/entrepreneurial mindset

Responsibilities

Build and maintain automated data validation tests using Databricks notebooks and tools like Pytest
Test data ingestion, transformation, and loading processes within the Databricks Lakehouse, specifically focusing on the Bronze, Silver, and Gold layers of the Medallion architecture
Implement tests for data accuracy, completeness, consistency, timeliness, and uniqueness at different points in the pipeline to catch data issues early
Reconcile data by comparing record counts, schemas, and values between source systems and target tables in Databricks
Implement automated data quality checks within data pipelines to ensure no data regressions occur with new code deployments
Implement automated monitoring and alerting for data quality metrics, identifying anomalies in data freshness, schema evolution, and volume
Work closely with data engineers and product owners to understand data requirements and ensure data quality meets business needs
Ensure compliance with data governance policies by building quality checks that validate data sensitivity, masking, and lineage, leveraging tools like Unity Catalog
Communicate project status and new discoveries in a clear and timely manner during daily stand-ups