Sr ML Engineer, Business Process Optimization

Workday • Pleasanton, CA

23h • $163,000 - $288,000 • Hybrid

About The Position

The Senior Software Engineer on Workday’s Business Process Optimize (BPO) team will lead the technical design, development, and scaling of machine learning pipelines, predictive analytics engines, and AI-driven recommendation features. BPO supports Workday’s enterprise AI and automation strategy, serving more than 100 Tier-0 customer accounts and handling millions of daily inference requests through high-throughput analytics, Model Context Protocol (MCP), and agentic workflows.

The role bridges production-grade distributed software engineering and modern ML/LLM architectures. It involves architecting low-latency inference services, building automated feature stores and retraining pipelines on big-data infrastructure, and extending AI observability across agent execution loops. The team is part of Admin Central, which is developing a unified experience for Workday administrators and building the platform infrastructure behind Business Process Optimize.

Requirements

• At least 8 years of experience researching, developing, and implementing advanced machine learning models and techniques, including at least 4 years building, deploying, and operating production ML systems at scale.
• At least 5 years of experience and proficiency in deep learning, NLP, and GenAI/LLM orchestration.
• Experience with frameworks and libraries such as PyTorch, TensorFlow, Scikit-learn, LangChain, and LlamaIndex, and proficiency in Java, Go, Scala, or C++ for high-performance backend systems.
• Strong ability with distributed processing technologies such as Spark, Flink, and Kafka, and high-throughput columnar or vector storage such as ClickHouse, Pinecone, or Milvus.
• Knowledge of RAG architectures, LLM fine-tuning, intent classification, and recommendation engines; ability to apply ML algorithms and Model-Based Design, including MCP tooling and structured agentic execution paths, to multi-agent orchestration, contextual reasoning, and workflow optimization.
• Experience leading exploratory data analysis of multi-tenant execution logs to validate prompt and response accuracy and optimize tool usage; strong feature engineering skills for vector embeddings, feature stores, and multi-tenant data isolation.
• Experience leading model building, including foundation model adaptation, prompt engineering, fine-tuning, and low-latency inference; understanding of the model development lifecycle and MLOps platforms such as Kubeflow, MLflow, Airflow, or SageMaker for monitoring, latency tracking, and retraining at scale.
• Strong proficiency in Python and ML libraries including PyTorch, TensorFlow, NumPy, Pandas, Scikit-learn, LangChain, LangSmith, and DSPy.
• Experience deploying scalable ML systems in cloud-native production using Docker, Kubernetes, gRPC/REST APIs, and CI/CD under high-availability (99.9% uptime) and low-latency SLAs.
• Ability to mentor team members, drive cross-functional technical alignment, draft system design specifications and model cards, and communicate technical and governance decisions.
• Bachelor’s degree in Computer Science, Data Science, Machine Learning, Applied Mathematics, Statistics, or Engineering.

Nice To Haves

• Master’s degree or PhD; these are strongly preferred.
• Experience with Model Context Protocol (MCP), LangSmith, and LangChain-based observability and agent workflows.
• Experience with ClickHouse, Apache Iceberg, Spark, Kafka, or Flink in big-data and feature-store environments.

Responsibilities

• Architect, deploy, and maintain end-to-end machine learning pipelines, from distributed data ingestion and feature engineering through model training, evaluation, and production deployment.
• Design algorithms and LLM-driven recommendation features, such as IDD insights and next-best-action models, to optimize agentic workflows and tool usage.
• Optimize real-time model inference and scoring microservices for sub-second responses in high-concurrency, multi-tenant environments.
• Integrate telemetry and tracing into ML pipelines to monitor model performance, prompt and response drift, and execution efficiency.
• Collaborate with backend engineers to use big-data platforms for high-dimensional feature retrieval and analytics.
• Partner with Product Managers, Data Engineering, SRE, and Legal/Compliance to translate business and governance requirements into scalable ML solutions.
• Lead technical design, development, and scaling of ML pipelines, predictive analytics engines, and AI-powered recommendation features.
• Mentor team members, drive cross-functional technical alignment, draft system design specifications and model cards, and communicate technical and governance decisions.

Benefits

• The role may be eligible for the Workday Bonus Plan or a role-specific commission/bonus.
• Annual refresh stock grants may be included in the total compensation package.
• Workday advertises comprehensive benefits, but specific benefit details are not included in the posting.
• Flexible scheduling through Workday’s Flex Work approach.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior