Databricks Platform Engineer

EXL•New York, NY
•$100,000 - $140,000

About The Position

The Databricks Platform Manager/Engineer is responsible for designing, administering, securing, and optimizing an enterprise Databricks platform on AWS. The role owns the platform lifecycle and ensures scalability, security, governance, cost optimization, reliability, and operational excellence, enabling data engineering, analytics, and AI teams to work efficiently. The focus is platform engineering rather than data engineering or analytics development. The position includes administering Databricks workspaces and compute, managing identity and access, establishing governance, supporting CI/CD and infrastructure automation, monitoring operations, and optimizing costs. It also includes supporting critical and nonproduction databases and applications in multitenant, high-stress environments, including installation, configuration, management, monitoring, and integration of databases and third-party tools in SOX- and PHI-compliant environments. The role develops procedures for automated monitoring and proactive intervention to prevent customer impact.

Requirements

  • 8–10 years of overall IT experience, including 5 or more years of hands-on experience administering and managing Databricks platforms in production on AWS.
  • Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field; the qualifications section also specifies a bachelor's degree or post-bachelor's degree and 8–10+ years.
  • Strong experience in system design and architecture, including architecture decision-making.
  • Strong AWS and DevOps experience.
  • Hands-on experience with Terraform and CI/CD pipelines.
  • Strong understanding of cloud networking, security, scalability, and high availability.
  • Production troubleshooting and problem-solving skills.
  • Hands-on experience managing enterprise Databricks platforms and setting up multiple workspaces.
  • Experience configuring Unity Catalog, integrating AWS IAM, creating cluster policies, managing runtime upgrades, implementing CI/CD and Git integration, and managing secrets and audit logging.
  • Experience with platform monitoring, troubleshooting, production support, and cost optimization initiatives.
  • Databricks skills: workspace administration, cluster management, cluster policies, instance pools, job clusters, Databricks Runtime management, workflows, Databricks Repos, Unity Catalog, secret scopes, service principals, and personal access tokens.
  • AWS skills: IAM, VPC fundamentals, EC2 concepts, S3, CloudWatch, KMS, security groups, and networking fundamentals.
  • DevOps skills: Git, CI/CD, Terraform, infrastructure as code, and Azure DevOps, GitHub Actions, or Jenkins.
  • Monitoring skills: platform monitoring, incident management, log analysis, performance tuning, and cost optimization.

Nice To Haves

  • Experience managing large-scale Databricks environments with 100 or more users.
  • Experience supporting multiple business units or enterprise data platforms.
  • Hands-on experience with Terraform for Databricks infrastructure provisioning.
  • Knowledge of Lakehouse architecture and Delta Lake.
  • Experience with FinOps practices for cloud cost optimization.
  • Familiarity with data governance and security frameworks.
  • Experience with AWS Organizations and Control Tower.
  • Knowledge of private connectivity networking concepts, including PrivateLink and VPC endpoints.
  • Experience with monitoring tools such as CloudWatch, Datadog, or Splunk.
  • Exposure to Apache Spark internals.
  • Experience supporting AI/ML workloads on Databricks.
  • Databricks Certified Data Engineer Professional or Databricks Certified Platform Administrator certification, if available.
  • AWS Certified Solutions Architect Associate or Professional certification.
  • AWS Certified SysOps Administrator certification.
  • Terraform Associate certification.

Responsibilities

  • Make system design and architecture decisions for the Databricks platform.
  • Design, deploy, and manage Databricks workspaces across Development, Test, UAT, and Production environments.
  • Establish workspace standards, networking, and environment isolation.
  • Plan, test, and roll out Databricks Runtime upgrades.
  • Maintain platform availability, reliability, and performance.
  • Create and manage cluster policies, instance pools, and job clusters; standardize configurations and optimize startup times and resource utilization.
  • Integrate Databricks with AWS IAM and manage authentication, authorization, service principals, personal access tokens, and secret scopes.
  • Implement least-privilege access controls and support enterprise identity management and SSO.
  • Administer Unity Catalog, including catalogs, schemas, tables, permissions, role-based access control, and data governance policies.
  • Support audit and compliance requirements, configure audit logging, and maintain operational documentation and platform standards.
  • Configure Git integration with Databricks Repos and support CI/CD pipelines for notebooks, workflows, and infrastructure.
  • Collaborate with DevOps teams on automated deployments and enable infrastructure as code using Terraform where applicable.
  • Monitor workspace health, jobs, clusters, and platform performance; maintain operational dashboards and alerts.
  • Respond to incidents, perform root cause analysis, analyze logs, troubleshoot production issues, and support production releases and maintenance.
  • Monitor Databricks usage and cloud spending; recommend cluster sizing and autoscaling strategies, optimize scheduling and compute utilization, track cost trends, and identify savings opportunities.
  • Support critical and nonproduction databases in multitenant, high-stress environments.
  • Install, configure, manage, and monitor databases and applications in a SOX- and PHI-compliant environment.
  • Install, configure, and integrate third-party applications and tools.
  • Develop procedures for automated monitoring and proactive intervention to prevent customer impact.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service