Databricks Platform Engineer

Agility Technologies Inc

2d•Remote

About The Position

The Databricks Platform Engineer will build, maintain, and optimize the enterprise data lakehouse infrastructure. The ideal candidate will have strong DevOps and data engineering skills, with expertise in automating Databricks platform operations, ensuring reliability, and enabling data teams to efficiently leverage the platform. You will be responsible for platform provisioning, monitoring, cost optimization, and providing technical support for Databricks workloads.

Requirements

5+ years of experience in platform engineering, cloud infrastructure, or IT operations, with 2+ years of hands-on experience managing Databricks environments.
Strong knowledge of Databricks administration, including cluster setup, workspace management, and job orchestration.
Proficiency in Apache Spark for distributed data processing.
Hands-on experience with cloud platforms (Azure Government, AWS GovCloud, or GCP Public Sector).
Proficiency in scripting and automation with Python, Bash, or PowerShell.
Experience with Infrastructure-as-Code (IaC) tools like Terraform, CloudFormation, or ARM templates.
Solid understanding of cloud networking concepts, such as VPCs, subnets, security groups, and private endpoints.
Familiarity with federal compliance frameworks, including FedRAMP, FISMA, NIST 800-53, and CMMC.
Experience implementing data security best practices, such as RBAC, encryption (at rest and in transit), and audit logging.
Ability to set up and use monitoring and logging tools (e.g., Datadog, AWS CloudWatch, or Azure Monitor) to ensure platform stability.
Proficiency in diagnosing and resolving cluster performance, job failures, and other technical issues.

Nice To Haves

Databricks Certified Associate or Platform Administrator certification
Knowledge of Unity Catalog administration and data governance implementation
Experience with cost optimization strategies for Databricks, including efficient resource allocation, autoscaling, and workload management.
Advanced knowledge of performance tuning for Databricks clusters and Spark jobs.
Implementation of fine-grained access controls and data masking to comply with federal data protection standards.
Experience implementing single sign-on (SSO), multi-factor authentication (MFA), and data retention policies for secure environments.
Hands-on experience building automation scripts for platform provisioning, monitoring, and maintenance.
Familiarity with automation frameworks and tools that integrate with cloud platforms (e.g., AWS Lambda, Azure Logic Apps, Google Cloud Functions).
Knowledge of ETL/ELT pipelines and experience integrating Databricks with data warehousing solutions like Redshift, Snowflake, or BigQuery.
Familiarity with real-time data streaming technologies, such as Kafka, AWS Kinesis, or Azure Event Hubs
Federal cloud certifications such as AWS Certified Solutions Architect – Associate/Professional, Azure Administrator Associate, or Google Cloud Associate Cloud Engineer.
Public Trust (or higher) clearance eligibility; prior federal client experience.

Responsibilities

Build and maintain Databricks workspace infrastructure across development, staging, and production environments
Design, implement, and maintain Databricks platforms in cloud environments (Azure, AWS, or GCP), ensuring scalability, reliability, and cost-efficiency.
Manage and optimize Databricks clusters, including autoscaling, resource allocation, monitoring, and troubleshooting.
Oversee the deployment and configuration of Databricks workspaces, including user management, access controls, and workspace policies.
Collaborate with infrastructure and DevOps teams to implement CI/CD pipelines for deploying Databricks workflows and notebooks.
Monitor and manage performance, logging, and system health of the Databricks platform.
Implement robust data security and governance measures in compliance with organizational policies and standards.
Work with cross-functional teams to troubleshoot and resolve platform-related issues.
Automate repetitive tasks, such as provisioning, upgrades, and scaling, using Infrastructure-as-Code (IaC) tools like Terraform or CloudFormation.
Enable data engineers and data scientists by providing support for integrating Databricks with various data sources and tools.
Stay up-to-date with the latest Databricks features, updates, and best practices to ensure the platform meets evolving business needs.