Digital Platforms, Senior Data Engineer

PCI Pharma Services•Philadelphia, PA

About The Position

The Senior Data Engineer (Databricks Platform) is a senior individual contributor responsible for the hands-on design, development, optimization, and operational support of PCI’s enterprise data platform built on Databricks and AWS. The role helps ensure scalable, secure, high-performance data pipelines, models, and integrations that support analytics, reporting, and advanced AI/ML use cases across the organization. The position calls for deep expertise in distributed data processing, Delta Lake architecture, data modeling, ETL/ELT frameworks, and AWS cloud environments. It emphasizes execution excellence, platform reliability, and technical problem resolution, with close collaboration across the Data Platform Lead, Product, Project Management, Testing, Customer Success, DevOps, QA, and external development partners. The role supports compliance with applicable PCI, cGMP, GCP, and regulatory policies. Occasional overtime and/or weekend work may be required for critical releases or production needs.

Requirements

Deep expertise in distributed data processing.
Expertise in Delta Lake architecture, data modeling, ETL/ELT frameworks, and AWS cloud environments.
Hands-on experience with Databricks, Spark, Delta Lake, PySpark, and Databricks SQL.
Hands-on experience with AWS Glue, S3, and Lambda.
Ability to design and maintain data models including 3NF, star schema, and canonical models, as appropriate.
Ability to support cloud operations, deployments, integrations, testing, and production needs.
Ability to collaborate across technical and business teams and translate business and operational requirements into technical solutions.
Adherence to applicable PCI, cGMP, GCP, and regulatory policies.

Responsibilities

Design, develop, and maintain scalable data pipelines using Databricks, Spark, and Delta Lake.
Build robust ETL/ELT workflows using PySpark and Databricks SQL.
Implement and support Bronze, Silver, and Gold layers in a medallion architecture.
Design and maintain efficient data models, including 3NF, star schema, and canonical models, as appropriate.
Support and maintain Data Catalog and Data Governance frameworks to ensure data quality and compliance.
Build and maintain CI/CD pipelines for reliable deployment across environments.
Collaborate with external partners on development activities and ongoing production support.
Provide operational support for AWS-hosted application services, data services, and system integrations.
Monitor cloud environments for availability, performance, security, and cost efficiency, escalating risks and issues to the Technical Lead when required.
Support deployments and ensure environment readiness in collaboration with DevOps and technical stakeholders.
Contribute to cloud operational best practices for scalability, resilience, security, and incident response.
Support UAT and validation activities by reviewing test cases for technical accuracy and risk coverage, triaging defects, and ensuring test environment readiness and stability.
Provide technical input during release-readiness discussions and risk evaluations.
Collaborate with Product Managers, Project Managers, QA, DevOps, and external development partners to support solution delivery and platform stability.
Translate business and operational requirements into scalable technical solutions aligned with approved architecture and enterprise standards.
Support secure and reliable integration with internal enterprise systems and data sources.
Follow applicable PCI, cGMP, GCP, and regulatory policies.
Maintain consistent attendance and reliable availability, and perform additional duties as assigned by the Data Platform Technical Lead or management.