Senior Architect, Databricks

West Monroe•Los Angeles, NY

3d•Hybrid

About The Position

West Monroe is seeking a Senior Architect to join its Technology and Experience Practice. The role works with clients to develop Big Data strategies and roadmaps, with an emphasis on Databricks, across industries such as healthcare, financial services, insurance, and utilities. The position combines data architecture and engineering with enterprise AI/ML and Generative AI solution design. It is hybrid, with one to two days per week in an office, and requires travel of up to 30–50%.

Requirements

At least 10 years of professional experience.
At least 3 years of experience with Databricks.
At least 8 years of prior consulting experience.
Strong technical expertise in Databricks architecture, performance optimization, Delta Lake, Unity Catalog, workflow orchestration, security, governance, and cloud-native data engineering best practices.
Experience architecting and deploying AI/ML or Generative AI solutions in enterprise environments.
Hands-on experience with LLM ecosystems and frameworks such as OpenAI, Azure OpenAI, LangChain, Semantic Kernel, MLflow, or Databricks Mosaic AI.
Knowledge of Retrieval-Augmented Generation (RAG), vector databases, embeddings, prompt engineering, and AI orchestration patterns.
Familiarity with AI governance, model lifecycle management, responsible AI principles, and enterprise security considerations.
Experience integrating AI capabilities into data platforms, analytics solutions, or customer-facing applications.
Knowledge of standard methodologies, concepts, best practices, and procedures in a Big Data environment.
Commitment to inclusion and diversity and openness to new ideas and perspectives.
Ability to meet travel requirements of up to 30–50%.

Responsibilities

Design data flows and processing pipelines for analysis using modern tools such as Spark on Scala, Storm, Flume, and Sqoop.
Design data structures for ingestion and reporting based on the use case and technology.
Advise clients on requirements, data architecture, platform selection, and platform design, including data structuring, stewardship, semantics and syntax of data attributes, coding structures, and mapping schemes.
Collaborate with data science and engineering teams to productionize machine learning and Generative AI workloads in modern cloud ecosystems.
Establish best practices for AI data readiness, feature engineering, model monitoring, observability, and governance.
Lead technical discussions on AI-enabled modernization strategies, including semantic layers, vector databases, AI agents, and knowledge retrieval systems.
Evaluate emerging AI technologies and recommend fit-for-purpose tools aligned with client objectives, scalability, security, and compliance requirements.
Guide and coach consultants, share best practices, and perform code reviews.
Coordinate with data source application owners to ensure integration between the analytics platform and source systems.
Ensure data integrity issues are addressed.
Attend and participate in meetings necessary to perform the role and support business and technical development.