Senior Architect, Databricks
About The Position
West Monroe is seeking a Senior Architect to join its Technology and Experience Practice. The role works with clients to develop Big Data strategy and roadmaps, with an emphasis on Databricks, across industries including healthcare, financial services, insurance, and utilities. It includes designing data architectures and pipelines, advising on data management and platform selection, and helping productionize AI and machine-learning workloads in cloud environments. The Senior Architect also leads technical discussions on AI-enabled modernization, evaluates emerging technologies, and guides consultants through coaching and code reviews. The position is hybrid, with 1 to 2 days per week in the office, and requires travel of up to 30% to 50%.
Requirements
- 10+ years of professional experience.
- 3+ years of experience with Databricks.
- 8+ years of prior consulting experience.
- Very strong technical expertise in Databricks, including architecture, performance optimization, Delta Lake, Unity Catalog, workflow orchestration, security, governance, and cloud-native data engineering best practices.
- Experience architecting and deploying AI/ML or Generative AI solutions in enterprise environments.
- Hands-on experience with LLM ecosystems and frameworks such as OpenAI, Azure OpenAI, LangChain, Semantic Kernel, MLflow, or Databricks Mosaic AI.
- Knowledge of Retrieval-Augmented Generation (RAG), vector databases, embeddings, prompt engineering, and AI orchestration patterns.
- Familiarity with AI governance, model lifecycle management, responsible AI principles, and enterprise security considerations.
- Experience integrating AI capabilities into data platforms, analytics solutions, or customer-facing applications.
- Knowledge of standard methodologies, concepts, best practices, and procedures in a Big Data environment.
- Commitment to inclusion and diversity, and openness to new ideas and perspectives.
- Willingness to travel up to 30% to 50%.
Responsibilities
- Design data flow and processing pipelines for analysis using modern toolsets such as Spark on Scala, Storm, Flume, and Sqoop.
- Design data structures for ingestion and reporting based on the use case and technology.
- Advise clients on requirements, data architecture, platform selection, and platform design, including data structure, stewardship, semantics and syntax of data attributes, coding structures, and mapping schemes.
- Collaborate with data science and engineering teams to productionize machine-learning and Generative AI workloads in modern cloud ecosystems.
- Establish best practices for AI data readiness, feature engineering, model monitoring, observability, and governance.
- Lead technical discussions on AI-enabled modernization strategies, including semantic layers, vector databases, AI agents, and knowledge retrieval systems.
- Evaluate emerging AI technologies and recommend tooling aligned with client objectives, scalability, security, and compliance requirements.
- Guide and coach consultants, share best practices, and perform code reviews.
- Coordinate with data-source application owners to ensure integration with analytics platforms and source systems.
- Ensure data integrity issues are addressed.
- Attend and liaise at meetings necessary to perform duties and support business and technical development.
Benefits
- Medical insurance for employees and their families.
- Dental insurance for employees and their families.
- Vision insurance for employees and their families.
- Basic life insurance for employees and their families.
- 401(k) plan.
- Employee stock ownership program with the ability to purchase shares.
- Eligibility for annual bonuses.
- Unlimited flexible time off.
- Ten paid holidays per calendar year.
- Eligibility for ten weeks of paid parental leave upon hire.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
