Data Engineer / Platform Administrator (Azure Databricks)

Vanderbilt University Medical Center•Nashville, TN

1d•Remote

About The Position

The Data Platform Services team is seeking a Data Engineer / Platform Administrator with an inquisitive, analytical mindset to deepen their technical skills. Approximately 80% of the role will involve administering the Azure Databricks platform, which includes over 100 workspaces. This administration will encompass building automation, monitoring performance and costs, managing configurations, testing new features, and promoting best practices. The remaining 20% of the time will be dedicated to data engineering tasks, such as building and optimizing data pipelines as part of the team. The role offers ownership of defined platform areas from the outset, with support from senior engineers and opportunities for expanded scope as the individual grows. This position can be performed fully remotely from most US locations.

Requirements

Bachelor's degree
1+ year of experience with Databricks
2+ years of Python in an object-oriented paradigm
2+ years implementing cloud solutions with Azure, AWS, or GCP
1+ year of SQL experience

Nice To Haves

Experience with Infrastructure as Code solutions such as Terraform
Experience with CI/CD automation in GitLab, GitHub, or Azure DevOps
Experience with REST data services, APIs, and microservices
Experience with containerization using Docker or Podman
Experience with PowerShell and/or Bash
A link to your GitHub profile or other personal code examples

Responsibilities

Administer Azure Databricks environment of 100+ workspaces, including compute policies, access controls, and governance, collaborating with senior team members on architectural decisions.
Build automation and Infrastructure as Code to provision and manage workspaces at scale.
Develop solutions for proactive monitoring of platform health, performance, and costs.
Test new Databricks features and promote best practices to teams across the organization.
Develop tools and services for other teams to create, test, and deliver data-related assets.
Design and develop performance-focused data pipelines supporting various source systems (flat files, APIs, databases, etc.).
Troubleshoot production pipeline issues and collaborate with internal stakeholders to deploy fixes.
Build reusable solutions for recurring problems and ensure data accuracy and integrity.
Identify data needs with stakeholders and seek opportunities to improve platform scalability.