Digital Platforms, Senior Data Engineer

PCI Pharma Services • Philadelphia, PA

2d

About The Position

The Senior Data Engineer (Databricks Platform) is a senior individual contributor responsible for the hands-on design, development, optimization, and operational support of the enterprise data platform built on Databricks and AWS technologies. This role ensures scalable, secure, and high-performance data pipelines, models, and integrations that power analytics, reporting, and advanced AI/ML use cases across the organization. The role brings deep expertise in distributed data processing, Delta Lake architecture, data modeling, ETL/ELT frameworks, and cloud environments (AWS). This role focuses on execution excellence, platform reliability, and technical problem resolution, while collaborating closely with Data Platform Lead, Product, Project Management, Testing, Customer Success, and external development partners.

Requirements

• Deep expertise in distributed data processing, Delta Lake architecture, data modeling, ETL/ELT frameworks, and cloud environments (AWS).
• Hands-on experience with AWS Glue, S3, and Lambda.
• Experience with Databricks (Spark / Delta Lake).
• Experience with PySpark and Databricks SQL.
• Experience with data modeling (3NF, star schema, canonical models).
• Experience with CI/CD pipelines.
• Experience supporting Data Catalog and Data Governance frameworks.
• Experience supporting AWS-hosted infrastructure.
• Experience monitoring cloud environments for availability, performance, security, and cost efficiency.
• Experience supporting deployment activities and ensuring environment readiness.
• Experience collaborating with Product Managers, Project Managers, QA, DevOps, and external development partners.
• Ability to translate business and operational requirements into technical solutions.
• Ability to support secure and reliable integration with internal enterprise systems and data sources.
• Adherence to all applicable PCI, cGMP, GCP, and regulatory policies.
• Consistent attendance and reliable availability.

Nice To Haves

• Experience with medallion architecture (Bronze, Silver, Gold layers).

Responsibilities

• Design, develop, and maintain scalable data pipelines using Databricks (Spark / Delta Lake).
• Build robust ETL/ELT workflows leveraging PySpark and Databricks SQL.
• Implement and support medallion architecture (Bronze, Silver, Gold layers).
• Design and maintain efficient data models, including 3NF, star schema, and canonical models, as appropriate.
• Support and maintain Data Catalog and Data Governance frameworks to ensure data quality and compliance.
• Build and maintain CI/CD pipelines to enable reliable deployment across environments.
• Collaborate with external partners for development activities and ongoing production support.
• Demonstrate hands-on experience with AWS Glue, S3, and Lambda.
• Provide operational support for AWS-hosted infrastructure, including application services, data services, and system integrations.
• Monitor cloud environments for availability, performance, security, and cost efficiency; escalate risks and issues to the Technical Lead when required.
• Support deployment activities and ensure environment readiness in collaboration with DevOps and technical stakeholders.
• Contribute to cloud operational best practices focused on scalability, resilience, security, and incident response.
• Partner with Testing and QA teams to support User Acceptance Testing (UAT) and validation activities by: Reviewing test cases for technical accuracy and risk coverage, Triaging defects identified during testing cycles, Ensuring test environment readiness and stability.
• Provide technical input during release readiness discussions and risk evaluations.
• Collaborate closely with Product Managers, Project Managers, QA, DevOps, and external development partners to ensure successful solution delivery and ongoing platform stability.
• Translate business and operational requirements into scalable technical solutions aligned with approved architecture and enterprise standards.
• Support secure and reliable integration with internal enterprise systems and data sources.
• Ensure adherence to all applicable PCI, cGMP, GCP, and regulatory policies.
• Participate in occasional overtime and/or weekend work as required to support critical releases or production needs.
• Maintain consistent attendance and reliable availability as essential functions of the role.
• Perform additional duties as assigned by the Data Platform Technical Lead or management.

Benefits

• Equity and Inclusion are at the core of our company’s purpose: Together, delivering life-changing therapies.
• We are committed to cultivating an inclusive workplace by holding ourselves accountable to the highest standards of understanding, fairness, respect, and equal opportunity – at every level.
• We envision a PCI community where everyone can belong and grow, and we strive to bring this vision to reality by continuously and intentionally assessing our people practices, policies and programs, marketing approach, and workplace culture.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed