Senior Data Platform Engineer

Arrowhead Pharmaceuticals•San Diego, CA

3d•$155,000 - $190,000

About The Position

The Senior Data Platform Engineer designs, builds, and manages Arrowhead Pharmaceuticals’ Databricks environments, taking end-to-end ownership of data platform infrastructure, security, and performance. The role leads ingestion of data from internal systems and external providers into the lakehouse, partners with business and analytics stakeholders to develop and support Genie dashboards, and manages Databricks infrastructure—including workspaces, clusters, Unity Catalog, and jobs—as code using Terraform. The position calls for production Databricks administration, Terraform experience, and data pipeline development at enterprise scale.

Requirements

Bachelor's degree in Computer Science, Data Engineering, Information Systems, or a related field, or equivalent practical experience.
At least 5 years of experience in data engineering or platform engineering, including extensive hands-on experience administering and building on Databricks, including clusters, jobs, Unity Catalog, Delta Lake, and SQL warehouses.
Demonstrated experience managing Databricks infrastructure as code using Terraform.
Strong proficiency in Python and SQL for building and optimizing data pipelines.
Experience designing and building ingestion pipelines from diverse internal and external sources, including APIs, SFTP, third-party vendor feeds, and streaming.
Working knowledge of cloud platforms such as AWS or Azure and associated data and storage services.
Experience supporting business intelligence or natural-language dashboarding tools, such as Databricks Genie.
Experience implementing Databricks data governance, security, and access controls, including Unity Catalog permissions, row- or column-level security, service principals, and secrets management.
Ability to explain pipeline and dashboard design decisions to non-technical stakeholders, including scoping Genie dashboard requirements with business users.

Nice To Haves

Databricks certifications, such as Databricks Certified Data Engineer Professional or Databricks Certified Associate/Professional Platform Administrator.
Experience with Databricks Genie and building natural-language or self-service analytics experiences for business users.
Experience integrating data from external providers or vendor APIs at enterprise scale.
Familiarity with CI/CD tools such as GitHub Actions, Azure DevOps, or GitLab CI for automated deployment of Terraform-managed infrastructure.
Experience with orchestration tools such as Databricks Workflows, Airflow, or dbt.
Experience building and supporting agentic workloads on Databricks, including managing model and tool access and governance through Unity Catalog AI Gateway.
Expertise in pharma or pharma-related data, such as clinical, R&D, regulatory, or commercial data, and associated compliance considerations.

Responsibilities

Design, build, and manage Databricks environments, including workspaces, clusters, Unity Catalog, and SQL warehouses, for enterprise analytics and data science workloads.
Develop and maintain Terraform modules and pipelines to provision, configure, and version-control Databricks infrastructure and related cloud resources.
Design and build lakehouse ingestion pipelines for internal systems and external data providers, including APIs, SFTP, batch files, and streaming sources.
Architect and maintain bronze, silver, and gold data layers using Delta Lake, ensuring data quality, lineage, and governance through Unity Catalog.
Partner with business and analytics users to design, build, and support Genie dashboards and enable self-service, natural-language data exploration.
Implement platform security and access controls in partnership with data stewards.
Collaborate with data engineers, analysts, data scientists, AI engineers, and cloud DevOps engineers on CI/CD pipelines, automated tests, and Terraform infrastructure-as-code.
Document platform architecture, ingestion pipelines, and operational runbooks.
Assist AI engineers by building and maintaining MCP connectors and other data and tool integrations for agentic workflows.