Databricks Practice Architect II

TEKsystems•Beaverton, OR

1d•$148,200 - $222,400•Remote

About The Position

Join our Data Modernization and Gen AI practice and take the lead across designing, building, and maintaining data pipelines and analytics solutions on the Databricks cloud platform and collaborate with data analysts and customer stakeholders to ensure the availability, scalability, and reliability of data. We are seeking a highly skilled and motivated Databricks Data and AI Practice Architect with at least 7 years of hands-on experience in data engineering and at least 3 years in AI/ML engineering. Join our Data Modernization practice and take the lead across designing, building, and maintaining data pipelines, advanced analytics and AI/ML solutions on the Databricks cloud platform and collaborate with data analysts, data scientists, ML engineers and customer stakeholders to ensure the availability, scalability, and reliability of data. The Individual • Consultative and personable data and AI engineering leader, who will provide advice and leading expertise on Databricks and have a deep understanding of cloud data services to become a key contributor to various large scale data projects. • Works on specific projects critical to our needs with opportunities to switch teams and projects as our fast-paced business grows and evolves. • Is versatile, displays leadership qualities and is enthusiastic to take on new challenges across the data stack.

Requirements

Bachelor’s or higher degree in Computer Science, Information Technology, or a related field.
10+ years industry experience building and supporting large-scale cloud systems.
Ability to speak and write in English fluently.
Excellent verbal and written communication skills.
Hands-on experience with Databricks, Spark and working knowledge of DBT for data processing and transformation.
Working understanding of Data Warehousing concepts of Data Vault, Dimensional modelling, OLAP design.
Hands-on experience building data pipelines using technology such as Azure Data Factory, AWS Glue, GCP Dataflow and Apache Spark (preferably in Databricks).
Extensive experience with CI/CD platforms such as GitLab CI, GitHub Actions, AWS CodeBuild, Azure Pipelines, GCP Cloud Build, and Jenkins.
Skilled in scripting languages including Python, SQL, PySpark, Jupyter Notebooks, R, Scala, Bash, and PowerShell.
Expert Level proficiency in: Databricks, PySpark
MLflow for model tracking and deployment
ML pipelines and associated MLOps tools
Experience in deploying ML models securely in cloud-native architectures
Experience in Generative AI Large Language Models (LLMs), Retrieval Augmented Generation (RAG), and Agentic AI techniques.
Experience applying or fine-tuning foundational models or large language models (LLM).
Demonstrated experience in designing and integrating diverse data sources to support Agentic AI systems, including structured, unstructured, and real-time data pipelines in production environments.
Proficiency with Retrieval-Augmented Generation (RAG) frameworks, including implementation of vector databases, embedding models, and retrieval mechanisms to enhance contextual relevance and model performance.
Experience in conducting exploratory data analysis (EDA), data cleaning, and validation.
Experience in engineering feature pipelines and automate feature stores within Databricks.
Experience in building end-to-end ML pipelines with MLflow for training, evaluation, and model lifecycle management.
Experience in providing guidance on implementing AI agent frameworks such as LlamaIndex, crewAI, LangGraph, and Azure AI Foundry.
Experience in designing, implementing, and optimizing AI solutions leveraging both Large Language Models (LLMs) and Small Language Models (SLMs), ensuring scalability, efficiency, and alignment with business objectives.
Experience in developing and deploying scalable Machine Learning models using Databricks and Lakehouse architecture.
Experience in utilizing development containers, unit testing/code quality review - standard best practices.
Strong expertise in version control systems.
Familiarity with logging and monitoring solutions.
Familiarity with containerization and orchestration technologies.
Strong problem-solving, troubleshooting, communication and collaboration skills.

Nice To Haves

Professional Certification in one or more areas: Cloud Architect, Solution Architect, Data Engineer, ML Engineer, Gen AI Engineer, Claude Code Architect
Databricks certified candidates will be preferred.

Responsibilities

Design & Architecture: Design and implement data and AI solutions for the Databricks platform and APIs. Work with data engineering and data science teams, customers, and other customer-facing teams. Actively Participate in whole AI/ML pipeline design, development, and implementation lifecycle.
Technical leadership: Provide technical guidance to customers on big data projects, including architectural design, data engineering, and model deployment
Customer collaboration: Work with customers to understand their requirements, and help them define solutions and a roadmap to meet their goals
Solution design: Create and review architecture and solution design artifacts
Technical expertise: Demonstrate expertise in Databricks Lakehouse architecture, delta lake medallion architecture, data pipelines, CI/CD pipelines etc.
Communication: Communicate complex technical concepts to non-technical stakeholders, such as business leaders and C-level executives
Documentation: Develop and maintain documentation for the Databricks platform, such as deployment guides, operational procedures, and architecture diagrams
Risk management: Identify, communicate, and mitigate risks, assumptions, issues, and decisions.