Sr. AI & Data Engineer - Databricks
About The Position
Accenture is hiring an experienced, hands-on Data & AI Engineer to design, build, and optimize scalable data pipelines using Databricks and Azure data services. The role includes designing and implementing autonomous agents with Databricks Mosaic AI Agent Framework and frameworks such as LangChain or AutoGen, and collaborating with analytics, data science, and cloud teams to provide reliable, high-performance data delivery for clients. The engineer will work across data modeling, data integration, data analytics, pipeline development, agentic AI, retrieval-augmented generation (RAG), cloud and infrastructure integration, data quality, observability, and governance. This is a hybrid role requiring three days per week in the office. Hiring locations are Dallas, Texas; Houston, Texas; Columbus, Ohio; and Tampa, Florida.
Requirements
- At least three years of professional experience as a Data & AI Engineer or in a related role.
- At least three years of experience working with Microsoft Azure cloud data services.
- At least two years of experience with Databricks, including PySpark, SQL, Delta Lake, and workflows.
- Bachelor's degree or equivalent (minimum 12 years) work experience; candidates with an associate degree must have at least six years of work experience.
- Ability to work in the office three days per week in this hybrid role.
- For U.S. employment, work authorization that does not now or in the future require visa sponsorship.
Nice To Haves
- Advanced Python (PySpark) and SQL for data transformation and pipeline development.
- Experience with Azure Databricks, including Spark, Delta Lake, Auto Loader, and Lakehouse Medallion Architecture (Bronze, Silver, Gold).
- Experience with Git-based version control and CI/CD pipelines, such as Azure DevOps or GitHub Actions.
- Experience in agent development using Mosaic AI Agent Framework.
- Strong problem-solving and performance optimization skills.
- Experience in MLOps or LLMOps, including CI/CD for ML and LLM lifecycles.
- Experience with the Genie product family.
- Databricks Certified Data Engineer Professional (2025).
- Microsoft Certified: Azure Data Engineer Associate.
Responsibilities
- Architect, develop, and maintain scalable, low-latency data pipelines to ingest and transform diverse datasets using Azure Databricks workflows and Spark technologies.
- Design and optimize Lakehouse architecture with Delta Lake tables and Bronze, Silver, and Gold layers for efficient, reliable, and governed data storage.
- Partner with data scientists and business analysts to deliver clean, validated, well-modeled datasets for advanced analytics and AI initiatives.
- Design and implement autonomous agents using Databricks Mosaic AI Agent Framework and frameworks such as LangChain or AutoGen.
- Build and optimize vector search indexes using Databricks Vector Search for accurate context retrieval.
- Process large-scale distributed datasets on the Databricks Intelligence Platform with Apache Spark.
- Design and implement data models that meet business requirements and improve data usability.
- Evaluate and improve data architecture processes to enhance efficiency and effectiveness.
- Design AI agents with retrieval, orchestration, tool invocation, evaluation harnesses, and lifecycle observability.
- Monitor pipeline health and performance and implement automated alerting and observability frameworks for troubleshooting and proactive optimization.
Benefits
- Medical coverage
- Dental insurance
- Vision insurance
- Life insurance
- Long-term disability coverage
- 401(k) plan
- Bonus opportunities
- Paid holidays
- Paid time off
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
