Senior CE Data Engineer

Eli Lilly and Company • Us, IN

1d • Onsite

About The Position

Lilly CIA powers the enterprise with governed, AI-ready data products, a converged semantic layer, and the identity and policy infrastructure behind Lilly's digital experiences. Within CIA, the LillyDirect Covered Entity (CE) Pharmacy pod builds and operates the data platform that sits inside the pharmacy's covered-entity trust boundary — handling identified PHI under HIPAA with strict isolation from parent-side systems. This role implements that platform on Databricks. The CE Data Engineer builds and operationalizes the CE Pharmacy data platform on Databricks — implementing the medallion pipelines, data contracts, identity crosswalk logic, and isolation controls that the CE Data Architect designs. This is a hands-on engineering role: writing production pipelines, evaluating and applying Databricks capabilities and integration patterns, and holding the line on data quality, testing, and monitoring so the platform is reliable, auditable, and built to scale. Works in close, day-to-day partnership with the CE Data Architect and the Associate Director of CE Data Engineering — turning architectural intent into running code — and with Analytics Engineers and the DataHub Product Owner to ship governed data products.

Requirements

• Bachelor's degree in Computer Science, Engineering, Information Technology, or similar degree.
• Experience in data engineering, with a focus on building production data pipelines and data products.
• Advanced SQL and Python; hands-on Databricks / Unity Catalog fluency (notebooks, Delta Lake, catalog/schema/grants, Workflows).
• Qualified applicants must be authorized to work in the United States on a full-time basis. Lilly will not provide support for or sponsor work authorization or visas for this role, including but not limited to F-1 CPT, F-1 OPT, F-1 STEM OPT, J-1, H-1B, TN, O-1, E-3, H-1B1, or L-1.

Nice To Haves

• Experience defining and executing data ingestion pipelines at enterprise scale.
• Working knowledge of data governance, classification, and access control (RBAC/ABAC, row- and column-level security).
• Proficiency with Git-based CI/CD workflows (Git Actions or equivalent) for versioned data and pipeline artifacts.
• Excellent problem-solving skills; ability to translate architectural designs into working, tested pipelines.
• Good communication and collaboration skills — able to work effectively with architects, product owners, and multi-functional partners.
• Experience in regulated or healthcare data environments; familiarity with HIPAA, PHI handling, and covered-entity constructs.
• Data contract frameworks (ODCS or comparable) and policy-as-code (OPA/Rego) awareness.
• Tokenization / de-identification pattern implementation (Datavant or comparable).
• PySpark and transformation-as-code (DLT); test frameworks (pytest, DQX / Great Expectations).
• Infrastructure-as-code (Terraform) for provisioning Unity Catalog objects and grants.
• MDM / identity resolution implementation experience (Reltio or comparable).
• Experience with Agile/Scrum methodologies and Jira.
• Experience building or integrating AI Skills and agents against governed data platforms.

Responsibilities

• Design and build scalable, efficient Databricks pipelines implementing the CE Architect's canonical data models across the medallion (bronze → silver → gold), scoped entirely within the CE trust boundary.
• Evaluate and apply Databricks capabilities and integration patterns — Unity Catalog, Delta Lake, Databricks Workflows, serverless compute, Lakebase, ingestion connectors — selecting the right tool for each pipeline given performance, cost, and scalability constraints.
• Implement the identity crosswalk and PHI classification logic defined by the CE Architect (Reltio MDM, Auth0/Passport, Datavant tokens, Scriptly, Genesys, Prescryptive, Transcend), safeguarding merge/split integrity in code.
• Build to ODCS data contracts — implement schema, quality, freshness/SLA, and lineage requirements as enforced pipeline logic, not documentation.
• Implement row/column-level security, masking, and tokenization boundaries so PHI isolation is enforced at the platform layer, in partnership with the Policy-as-Code Engineer's OPA/Rego policies.
• Own end-to-end pipeline development lifecycle — requirements to prototyping to production deployment and maintenance — for CE data products, in partnership with Analytics Engineers and the DataHub Product Owner.
• Build and maintain CI/CD pipelines (Github Actions [CI], Git-based promotion dev → test → prod) for CE data, contract, policy, and agent artifacts.
• Automate data ingestion and product creation to reduce manual pipeline maintenance and onboarding time for new CE data sources.
• Partner with the CE Data Architect on reference architecture and patterns, providing implementation feedback that keeps designs buildable and performant at scale.
• Build the data pipelines that lets CE Skills and agents consume governed data — ensuring PHI classification and consent travel with the data into agentic consumption paths.

Benefits

• company bonus (depending, in part, on company and individual performance)
• company-sponsored 401(k)
• pension
• vacation benefits
• medical, dental, vision and prescription drug benefits
• flexible benefits (e.g., healthcare and/or dependent day care flexible spending accounts)
• life insurance and death benefits
• certain time off and leave of absence benefits
• well-being benefits (e.g., employee assistance program, fitness benefits, and employee clubs and activities)

What This Job Offers

• Job Type: Full-time
• Career Level: Senior