Azure Databricks Administrator

NYU Langone Health•New York, NY

6d•$101,494 - $155,000

About The Position

We have an exciting opportunity to join our team as a Azure Databricks Administrator. As an Azure Databricks Administrator within this department, you will be responsible for managing and maintaining the Databricks platform, ensuring optimal performance and security across workspaces, clusters, and jobs. This role is crucial in overseeing user administration, managing Azure storage solutions, and implementing CI/CD pipelines using Azure DevOps. By leveraging a deep understanding of Databricks architecture and proficiency in scripting languages, the administrator will automate tasks and enhance the efficiency of data operations. Strong communication skills and a commitment to high-quality support will enable effective collaboration with cross-functional teams, directly contributing to the department's mission of delivering robust data solutions and insights.

Requirements

Bachelor's degree in Computer science, Information systems, or Engineering
7+ years of related professional experience administering data platforms
3+ years experience with Azure Databricks Lakehouse Platform
Proficiency in scripting languages such as Python/PySpark, PowerShell, or Bash
Experience in automating administrative tasks and workflows
Knowledge of security best practices and compliance requirements
Experience with ETL processes, data pipelines, and big data technologies
Experience with backup and restore procedures for Databricks and Azure services
Ability to troubleshoot and resolve issues in a timely and efficient manner
Strong verbal and written communication skills
Ability to document processes, procedures, and configurations clearly
Commitment to providing high-quality support to internal/external users and stakeholders
Ability to understand and address the needs of internal and external customers
Team player with the ability to work collaboratively with cross-functional teams
Flexibility to adapt to changing requirements and priorities
Willingness to learn new technologies and continuously improve skills

Nice To Haves

Deep understanding of Databricks architecture, features, and best practices
Experience with Delta Lake and Databricks SQL for advanced data management
Experience in performance tuning and optimization for both Databricks and Azure environments
Experience supporting/administering data science and ML workloads in Databricks
Knowledge of disaster recovery planning and implementation
Experience integrating Databricks with other data platforms and tools
Understanding of hybrid and multi-cloud environments
Experience implementing data protection measures and ensuring regulatory compliance

Responsibilities

Manage and maintain Azure Databricks Platform, workspaces, clusters, and jobs
Oversee user administration including access controls and permissions
Handle library installations, runtime management, and policy enforcement
Implement/analyze cost control measures and monitor platform usage and performance
Administer Unity Catalog for data governance and security
Collaborate with data engineers, data scientists, and analysts to optimize and streamline data workflows, and analytical pipelines on the Databricks platform
Manage Azure storage solutions, including Blob Storage and Data Lake Storage
Administer Azure Key Vault for secure storage of secrets and keys
Configure and manage Azure Data Factory for data integration and ETL processes
Implement and manage VNETs, firewalls, Azure policies and security best practices
Implement and manage Azure DevOps for CI/CD pipelines and version control
Implement best practices for automated deployment and continuous integration
Set up budgets and alerts to monitor and control Azure costs and spend
Implement monitoring solutions for platform health and performance
Configure alerts for proactive issue detection and resolution
Databricks Lakehouse Monitoring develop and execute ad hoc operational reports not already implemented within the EDC (e.g., nonstandard query reports).