Databricks Administrator

Capgemini • Seattle, WA

10d • $65,000 - $169,000

About The Position

Capgemini is urgently seeking a Databricks Administrator who is responsible for provisioning Databricks and Unity Catalog services while taking care of organizational policies and governance requirements as well as streamlining and automating data-related processes. Your Role:
Implement data provisioning patterns based on business requirements, following follow predefined processes, policies, standards, and metadata management rules
Create and manage distributed workspaces in Databricks, set up workspace policies, provision Databricks clusters and manage data infrastructure sizing and capacity
Create Python notebooks, implement data masking processes, create UDFs (SQL/Python), troubleshoot data pipelines
Ensure data security and compliance with regulations using Databricks and Privacera's features
Navigate multi-step enterprise approval process across architecture, security, and governance teams
Design and implement data architecture leveraging technologies such as Databricks, Unity Catalog, Privacera, and Collibra
Develop, optimize, and manage data pipelines for ETL processes using Databricks, with a focus on data integrity and quality
Design and maintain data models and schemas, incorporating Unity Catalog and Collibra data governance practices
Operationalize Machine Learning models in Batch and Real Time Data Pipelines, leveraging relevant governance setups
Collaborate with cross-functional teams including data scientists, engineers, and analysts to translate business requirements into scalable solutions

Requirements

• Account and Workspace Administration
• Configuration of workspaces across environments
• Management and optimization of compute resources
• Strong knowledge of security and governance: user and group management, provisioning, identify federation
• Role based access control, cluster policies and workspace object permissions
• Unity Catalog administration of metastores, catalogs, schemas, external locations and Delta sharing)
• Knowledge of security compliance procedures
• Strong engineering knowledge to guide data engineering teams on ways to build pipelines and optimize existing ones
• Usage monitoring and troubleshooting bottlenecks with spark jobs, ETL pipelines and ML workloads
• Scripting experience in Python and SQL
• Data sharing knowledge and implementing guardrails

Responsibilities

• Implement data provisioning patterns based on business requirements, following follow predefined processes, policies, standards, and metadata management rules
• Create and manage distributed workspaces in Databricks, set up workspace policies, provision Databricks clusters and manage data infrastructure sizing and capacity
• Create Python notebooks, implement data masking processes, create UDFs (SQL/Python), troubleshoot data pipelines
• Ensure data security and compliance with regulations using Databricks and Privacera's features
• Navigate multi-step enterprise approval process across architecture, security, and governance teams
• Design and implement data architecture leveraging technologies such as Databricks, Unity Catalog, Privacera, and Collibra
• Develop, optimize, and manage data pipelines for ETL processes using Databricks, with a focus on data integrity and quality
• Design and maintain data models and schemas, incorporating Unity Catalog and Collibra data governance practices
• Operationalize Machine Learning models in Batch and Real Time Data Pipelines, leveraging relevant governance setups
• Collaborate with cross-functional teams including data scientists, engineers, and analysts to translate business requirements into scalable solutions

Benefits

• Paid time off based on employee grade (A-F), defined by policy: Vacation: 12-25 days, depending on grade, Company paid holidays, Personal Days, Sick Leave
• Medical, dental, and vision coverage (or provincial healthcare coordination in Canada)
• Retirement savings plans (e.g., 401(k) in the U.S., RRSP in Canada)
• Life and disability insurance
• Employee assistance programs
• Other benefits as provided by local policy and eligibility

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed