Sr. Databricks Engineer
About The Position
Senior individual contributor responsible for designing, developing, enhancing, and supporting enterprise cloud data platform capabilities. The role reports to the Sr. Director of Data Platform Transformations & Operations, serves as a technical lead for major initiatives, and is accountable for production support and operational excellence. The work is approximately 70% new development and 30% operations, support, and platform reliability. Primary technologies include Azure Databricks, PySpark, SQL, Delta Lake, and cloud data engineering. The position is onsite in Austin, TX, and is limited to local candidates. Candidates must attend an onsite interview and confirm their willingness and availability.
Requirements
- Bachelor's degree in Computer Science, Information Systems, Engineering, Data Science, or a related field.
- At least 10 years of experience in data engineering, warehousing, or platform engineering.
- At least 5 years designing or implementing cloud-based data platforms.
- At least 3 years of hands-on Azure Databricks experience in enterprise environments.
- Advanced SQL, Python, PySpark, and Spark optimization skills.
- Experience with Databricks agentic AI, Delta Lake, Unity Catalog, Databricks Workflows, and Delta Live Tables.
- Deep understanding of lakehouse concepts, Medallion architecture, data modeling, and analytics data structures.
- Experience with Azure DevOps, Git, CI/CD, and automated testing and deployment.
- Experience implementing data quality, validation, governance, and security controls.
- Strong leadership, solution design, troubleshooting, and communication skills.
- Must be a local candidate able to work onsite in Austin, TX and attend an onsite interview.
Nice To Haves
- Master's degree.
- Experience in regulated environments, such as financial services, banking, or audit-sensitive settings.
- Databricks Data Engineer certification.
- Financial services industry experience.
- GenAI project deployment experience.
- Data governance expertise.
- Familiarity with Power BI, MicroStrategy, MDM, streaming architectures, infrastructure automation, or AI-assisted development.
- Experience with Databricks Asset Bundles.
Responsibilities
- Design, develop, and maintain scalable data pipelines, ingestion frameworks, and transformation processes.
- Implement Medallion (Bronze/Silver/Gold) architecture for reporting, analytics, AI, and self-service data.
- Develop and operationalize AI agent solutions on Databricks, integrating models, tools, and APIs.
- Build reusable frameworks, utilities, and platform components to improve productivity and quality.
- Develop and modernize batch, near-real-time, and streaming integration solutions.
- Serve as technical lead by providing solution design, conducting technical reviews, recommending tools, and overseeing implementation.
- Partner with architects to ensure solutions are scalable, secure, governed, and cost-effective.
- Mentor engineers and promote coding, testing, and documentation standards.
- Implement data quality, validation, reconciliation, monitoring, and security controls.
- Build and maintain CI/CD pipelines, automated testing, and release processes using Azure DevOps and Git.
- Contribute to platform observability, alerting, monitoring, performance tuning, and cost optimization.
- Participate in production support, incident response, troubleshooting, and root cause analysis.
- Create and maintain runbooks, support documentation, and knowledge-sharing resources.
- Collaborate with cross-functional teams to translate requirements into platform solutions.
- Support technical discovery, estimation, planning, delivery, and evaluation of new cloud, data, and AI capabilities.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
