Data Consultant (Databricks)
About The Position
Verinext has joined forces with Arctiq, combining our technical expertise and shared commitment to delivering innovative infrastructure and cloud solutions. As part of this expanded organization, we’re growing our team and looking for skilled professionals ready to solve complex challenges at scale. Arctiq is looking for a Data Engineer to lead the development of scalable data pipelines within the Databricks ecosystem. You will be responsible for architecting robust ETL/ELT processes using a "configuration-as-code" approach, ensuring our data lake house is governed, performant, and production-ready. Experience in migrating data ingestion and transformation workloads to Databricks using Lake flow declarative pipelines is key.
Requirements
- Strong hands-on experience with Databricks in production environments
- Deep expertise in PySpark and advanced SQL
- Experience with: Delta Lake ingestion pipelines (batch + streaming) data transformation frameworks/patterns
- Proven experience implementing: CI/CD in Databricks Databricks Asset Bundles (DABs) declarative pipelines (Lakeflow)
- Strong AWS infrastructure familiarity (S3, IAM, compute patterns)
- Terraform experience specifically with Databricks + AWS resources
- PowerShell scripting experience (asset)
Responsibilities
- Design and implement declarative, scalable pipelines using Lakeflow and Databricks Asset Bundles (DABs)
- Establish reusable pipeline templates and patterns aligned with CI/CD best practices
- Build a “configuration-as-code” approach for onboarding new sources and transformations quickly
- Develop high-volume ingestion pipelines using Databricks AutoLoader
- Implement CDC patterns (incremental loads, merges, deduping, late-arriving data)
- Ensure ingestion is resilient, observable, and optimized for cost/performance
- Configure and manage Unity Catalog for: metadata management access control / RBAC lineage and governance workflows
- Help enforce standards for data quality, naming, and environment separation
- Build and maintain complex workflows using Databricks Workflows / Jobs
- Integrate with external orchestration tools when required
- Improve operational reliability (retry logic, alerting, dependency handling)
- Use Terraform to manage Databricks workspace resources and AWS components
- Support AWS-aligned deployment patterns for: S3 storage compute configuration workspace setup and environment parity
Benefits
- Retirement Plan (401k, IRA)
- Work From Home
- Health Care Plan
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Education Level
No Education Listed
