Lead Software Engineer (Databricks Platform Engineer)
About The Position
In this role, you will lead the design, engineering, and evolution of the enterprise Databricks platform as a strategic capability that enables analytics, data products, machine learning, and AI solutions across the organization. You will develop reusable platform services, automation frameworks, reference architectures, and self-service capabilities that accelerate delivery while aligning with enterprise architecture, governance, and engineering standards. You will collaborate with cross-functional teams to enhance developer experience, drive platform adoption, evaluate emerging technologies, and deliver scalable, secure, and cost-effective healthcare data and AI solutions.
Requirements
- 7 or more years of relevant experience required.
- 3 or more years of hands-on experience designing, implementing, and supporting Databricks platform solutions required.
- Strong experience with Azure cloud services, cloud-native architectures, and enterprise platform engineering required.
- Experience with Terraform, Infrastructure-as-Code, and CI/CD pipelines, and infrastructure automation.
- Strong knowledge of Delta Lake, Unity Catalog, Apache Spark, and lakehouse architectures.
- Experience developing reusable platform services, automation frameworks, self-service capabilities, and engineering accelerators.
Nice To Haves
- Relevant degree preferred.
- Advanced degree in Computer Science, Engineering, Information Systems, or a related field preferred.
- Experience supporting machine learning and AI platform capabilities, including MLflow, Model Registry, Feature Store, Vector Search, or MLOps practices preferred.
- Databricks certification, experience with streaming and event-driven architectures, enterprise-scale data platforms, or healthcare industry experience preferred.
Responsibilities
- Lead the design, implementation, and continuous evolution of the enterprise Databricks platform and supporting architecture.
- Develop reusable platform services, reference architectures, engineering standards, and implementation patterns for analytics, data products, streaming, and AI workloads.
- Build Infrastructure-as-Code, CI/CD pipelines, and automation frameworks to improve platform scalability, reliability, and operational efficiency.
- Create self-service capabilities, reusable templates, and engineering accelerators that simplify platform onboarding and deployment.
- Optimize platform performance, scalability, security, governance, and cost efficiency across cloud environments.
- Enable machine learning and AI platform capabilities by implementing patterns for MLflow, Model Registry, Feature Store, Vector Search, and other Databricks technologies.
- Partner with Enterprise Architecture, Data Engineering, Product Management, Cloud Engineering, Data Science, and client stakeholders to deliver enterprise platform capabilities.
- Evaluate emerging Databricks features and cloud technologies, lead proof-of-concepts, and recommend platform enhancements that support business objectives.
- Provide technical leadership, mentoring, and engineering guidance while promoting Platform-as-a-Product principles and engineering best practices.
- Drive continuous improvement through automation, innovation, and adoption of enterprise platform standards.
Benefits
- Comprehensive benefits plan
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
