Data Engineering Manager

Sentara Hospitals•Virginia Beach, VA

17h

About The Position

Overview The Data Engineering Manager is responsible for Design and build a metadata-driven file ingestion framework on Databricks aligned with medallion architecture Lead onboarding of datasets using pattern-based ingestion (not one-off pipelines) Define and enforce data contracts including schema, keys, and data quality rules Implement scalable ingestion patterns supporting header / no-header / schema evolution scenarios Drive table-driven configuration model to eliminate dependency on static YAML-based onboarding Integrate ingestion framework with governance tools (e.g., DataHub) for lineage, discovery, and ownership Ensure strong data quality, auditability, and observability through centralized logging and control tables Collaborate with business and source system teams to understand data semantics and define keys Establish best practices for file ingestion, schema management, and incremental processing Lead vendor teams to ensure delivery aligns with target architecture and platform standards Education BLD - Bachelor's Level Degree in computer science required 15 years of experience in a complex computing environment may be considered in lieu of degree. Certification/Licensure Databricks Certification (Preferred) Experience 8+ years in data engineering with strong experience in modern data platforms (Azure preferred) Hands-on expertise in Databricks and PySpark for large-scale data processing Strong understanding of medallion architecture (Bronze / Silver / Gold) Experience designing metadata-driven or configuration-driven data pipelines Deep knowledge of data ingestion patterns (batch, file-based, incremental loads) Strong experience with data modeling concepts (keys, SCD, merge strategies) Experience implementing data quality frameworks and validation rules Proficiency in SQL and distributed data processing concepts Experience with Azure Data Lake Storage (ADLS) and file-based ingestion patterns Familiarity with CI/CD, Git, and deployment practices in data engineering Preferred Experience Experience with data catalog and governance tools (DataHub, Collibra, Alation, etc.) Exposure to customer 360 / MDM / CRM data integration (Salesforce, Dynamics, etc.) Experience working in healthcare or regulated environments (HIPAA, PHI handling) Familiarity with Unity Catalog and data access governance models Experience integrating data platforms with workflow tools like ServiceNow for access management Exposure to event-driven ingestion patterns (file triggers, streaming, etc.) Experience working in large-scale migration or modernization programs ,enterprise data platforms Experience enabling self-service data access and governance Familiarity with API-driven data integration patterns We provide market-competitive compensation packages, inclusive of base pay, incentives, and benefits. The base pay rate for Full Time employment is:$116,729.60-$216,777.60. Additional compensation may be available for this role such as shift differentials, standby/on-call, overtime, premiums, extra shift incentives, or bonus opportunities. Benefits: Caring For Your Family and Your Career Medical, Dental, Vision plans Adoption, Fertility and Surrogacy Reimbursement up to $10,000 Paid Time Off and Sick Leave Paid Parental & Family Caregiver Leave Emergency Backup Care Long-Term, Short-Term Disability, and Critical Illness plans Life Insurance 401k/403B with Employer Match Tuition Assistance – $5,250/year and discounted educational opportunities through Guild Education Student Debt Pay Down – $10,000 Reimbursement for certifications and free access to complete CEUs and professional development Pet Insurance Legal Resources Plan Colleagues have the opportunity to earn an annual discretionary bonus if established system and employee eligibility criteria is met. Sentara Health is an equal opportunity employer and prides itself on the diversity and inclusiveness of its close to an almost 30,000-member workforce. Diversity, inclusion, and belonging is a guiding principle of the organization to ensure its workforce reflects the communities it serves. In support of our mission “to improve health every day,” this is a tobacco-free environment. For positions that are available as remote work, Sentara Health employs associates in the following states: Alabama, Delaware, Florida, Georgia, Idaho, Indiana, Kansas, Louisiana, Maine, Maryland, Minnesota, Nebraska, Nevada, New Hampshire, North Carolina, North Dakota, Ohio, Oklahoma, Pennsylvania, South Carolina, South Dakota, Tennessee, Texas, Utah, Virginia, Washington, West Virginia, Wisconsin, and Wyoming.

Requirements

BLD - Bachelor's Level Degree in computer science required 15 years of experience in a complex computing environment may be considered in lieu of degree.
8+ years in data engineering with strong experience in modern data platforms (Azure preferred)
Hands-on expertise in Databricks and PySpark for large-scale data processing
Strong understanding of medallion architecture (Bronze / Silver / Gold)
Experience designing metadata-driven or configuration-driven data pipelines
Deep knowledge of data ingestion patterns (batch, file-based, incremental loads)
Strong experience with data modeling concepts (keys, SCD, merge strategies)
Experience implementing data quality frameworks and validation rules
Proficiency in SQL and distributed data processing concepts
Experience with Azure Data Lake Storage (ADLS) and file-based ingestion patterns
Familiarity with CI/CD, Git, and deployment practices in data engineering

Nice To Haves

Databricks Certification (Preferred)
Experience with data catalog and governance tools (DataHub, Collibra, Alation, etc.)
Exposure to customer 360 / MDM / CRM data integration (Salesforce, Dynamics, etc.)
Experience working in healthcare or regulated environments (HIPAA, PHI handling)
Familiarity with Unity Catalog and data access governance models
Experience integrating data platforms with workflow tools like ServiceNow for access management
Exposure to event-driven ingestion patterns (file triggers, streaming, etc.)
Experience working in large-scale migration or modernization programs ,enterprise data platforms
Experience enabling self-service data access and governance
Familiarity with API-driven data integration patterns

Responsibilities

Design and build a metadata-driven file ingestion framework on Databricks aligned with medallion architecture
Lead onboarding of datasets using pattern-based ingestion (not one-off pipelines)
Define and enforce data contracts including schema, keys, and data quality rules
Implement scalable ingestion patterns supporting header / no-header / schema evolution scenarios
Drive table-driven configuration model to eliminate dependency on static YAML-based onboarding
Integrate ingestion framework with governance tools (e.g., DataHub) for lineage, discovery, and ownership
Ensure strong data quality, auditability, and observability through centralized logging and control tables
Collaborate with business and source system teams to understand data semantics and define keys
Establish best practices for file ingestion, schema management, and incremental processing
Lead vendor teams to ensure delivery aligns with target architecture and platform standards

Benefits

Medical, Dental, Vision plans
Adoption, Fertility and Surrogacy Reimbursement up to $10,000
Paid Time Off and Sick Leave
Paid Parental & Family Caregiver Leave
Emergency Backup Care
Long-Term, Short-Term Disability, and Critical Illness plans
Life Insurance
401k/403B with Employer Match
Tuition Assistance – $5,250/year and discounted educational opportunities through Guild Education
Student Debt Pay Down – $10,000
Reimbursement for certifications and free access to complete CEUs and professional development
Pet Insurance
Legal Resources Plan
Colleagues have the opportunity to earn an annual discretionary bonus if established system and employee eligibility criteria is met