Azure Databricks consultants
About The Position
We are seeking a highly skilled Senior Databricks & AI Engineer to support the continued evolution of the Ask Anna platform. This position is critical to the success of the Ask Anna Performance Enhancement Initiative and will focus on improving platform performance, scalability, and AI capabilities based on production feedback. The ideal candidate will possess deep expertise in Databricks, Data Engineering, DataOps, Azure, APIs, and AI/GenAI technologies. This role will be instrumental in optimizing response times, enhancing the user experience, expanding data sources, and helping drive the future AI roadmap for the organization. This is a strategic position with significant visibility and impact, supporting the next generation of enterprise AI solutions.
Requirements
- 5+ years of experience in Data Engineering, Analytics Engineering, or related technical roles.
- Advanced hands-on experience with Databricks development, optimization, and administration.
- Strong experience with Data Engineering and modern data pipeline architectures.
- Proven experience implementing DataOps practices and automation.
- Experience working with Databricks Genie and AI-enabled analytics capabilities.
- Strong knowledge of Microsoft Azure services and cloud-native architectures.
- Experience designing and developing RESTful APIs and data integration services.
- Hands-on experience building, deploying, or supporting AI/GenAI solutions.
- Strong proficiency in SQL, Python, and modern data engineering frameworks.
- Excellent problem-solving and performance tuning skills.
- Databricks Performance Optimization (Must Have)
- Deep expertise in Databricks architecture, Spark optimization, query tuning, cluster management, and performance troubleshooting.
- Data Engineering & DataOps Strong experience building and optimizing data pipelines, ETL/ELT processes, data modeling, orchestration, monitoring, and CI/CD practices.
- AI / GenAI Solutions Experience building and deploying GenAI applications, RAG solutions, LLM integrations, Databricks Genie, and AI-enabled business solutions.
- Azure Cloud & API Development Hands-on experience with Azure services, cloud architecture, API design, and integrations between applications and Databricks.
Nice To Haves
- Experience with Angular front-end development.
- Experience developing AI Agents and autonomous workflows.
- Knowledge of Google Cloud Platform (GCP) services and AI offerings.
- Full-stack application development experience.
- Experience integrating AI solutions into business applications and enterprise workflows.
- Familiarity with conversational AI, semantic search, vector databases, or retrieval-augmented generation (RAG) architectures.
Responsibilities
- Lead Databricks performance tuning and optimization initiatives.
- Analyze and resolve performance bottlenecks impacting Ask Anna response times.
- Implement scalable data processing and query optimization strategies.
- Monitor platform performance and recommend continuous improvements.
- Design, develop, and maintain robust data pipelines within Databricks.
- Support DataOps best practices, automation, monitoring, and deployment processes.
- Improve data quality, reliability, and operational efficiency across the platform.
- Enable onboarding of new datasets and data sources into Ask Anna.
- Develop and optimize AI and GenAI-enabled capabilities within the Ask Anna ecosystem.
- Collaborate with stakeholders to identify opportunities for AI-driven enhancements.
- Support the development and deployment of AI-powered agents and intelligent workflows.
- Contribute to enterprise AI platform expansion and future innovation initiatives.
- Support Azure-based architecture, services, and AI solutions.
- Design and build APIs that enable seamless integration between front-end applications and Databricks services.
- Partner with application development teams to ensure scalable, secure, and high-performing solutions.
- Participate in architecture reviews and long-term technology planning.
- Contribute to the Ask Anna product roadmap and long-term performance strategy.
- Evaluate emerging Databricks, AI, and cloud technologies for future adoption.
- Support enterprise scalability initiatives and future AI platform capabilities.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
