Sr. Software Development Engineer / Principal, Software Development Engineer
About The Position
Workday is hiring a Senior or Principal Software Development Engineer for its AI Model Serving team. The team operates the services behind production AI workloads at Workday, including a gateway to vendor-hosted LLMs on GCP and AWS Bedrock, a model deployment platform, and Workday’s production model registry. It serves thousands of traditional ML models on sharded Ray Serve clusters, handling approximately 2,000 requests per second and more than 10,000 requests per second at peak in its largest cluster. It also provides a uniform interface for accessing models on Bedrock and Gemini. The engineer will help shape the platform’s vision and direction with the engineering manager, make critical technical and design decisions, and guide the production model registry’s evolution. The role focuses on building and operating scalable infrastructure for traditional ML models and LLMs, supporting Workday’s production AI and agents. Upcoming team priorities include strengthening the vendor interface, scaling for more than 20 production agents, improving cost governance, exploring hosting open-source LLMs, and evaluating performance-related service changes. The Senior level requires at least 6 years of related software development experience; the Principal level requires at least 10 years. Both levels require a bachelor’s degree in a relevant technical field or equivalent practical experience, along with deep distributed-systems and Python expertise, familiarity with production ML and LLM serving, monitoring capabilities, strong communication, and collaborative engineering skills.
Requirements
- For the Senior Software Development Engineer level: 6 or more years of related software development experience, focused on building and operating large-scale distributed systems.
- For the Principal Software Development Engineer level: 10 or more years of related software development experience, focused on building and operating large-scale distributed systems.
- Bachelor’s degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience.
- Deep experience designing, building, and scaling production-grade distributed systems.
- Understanding of the full software development lifecycle, including coding standards, testing, code reviews, source control management, and deployment.
- Deep proficiency in Python, with extensive experience writing production-level code and building systems using Python-based frameworks.
- Familiarity with LLMs and traditional machine learning models, including serving, scaling, and monitoring them in production, and designing platform abstractions for both.
- Ability to design and maintain monitoring strategies that provide insight into system health, performance, and cost.
- Excellent written and verbal communication skills, including writing clear design documents, explaining complex technical ideas, and building consensus across teams.
- Collaborative engineering approach, with experience mentoring engineers and fostering an inclusive team environment.
Responsibilities
- Lead the team technically and make design decisions that improve platform performance, reliability, and scalability.
- Design, implement, and maintain large-scale systems that move machine learning models into production.
- Write design documents to build consensus for new system components and enhancements to existing components.
- Evaluate and adopt technologies available within Workday and across the industry.
- Troubleshoot, improve, and scale continuous integration software pipelines.
- Build relationships with software engineers, machine learning engineers, and data scientists on partner teams.
- Respond to alerts and debug production issues to maintain platform health and reliability.
- Review pull requests and uphold consistency, performance, readability, and security across codebases.
- Develop documentation to share knowledge with other engineers.
- Help shape the AI Model Serving platform’s product vision in partnership with the engineering manager (Principal level).
- Guide the evolution of Workday’s production model registry to meet the needs of ML teams across the organization.
Benefits
- May be eligible for the Workday Bonus Plan or a role-specific commission/bonus.
- May be eligible for annual refresh stock grants.
- Workday provides a comprehensive benefits package; specific benefits are not listed in the posting.
- Flexible Work approach combines in-person and remote work, with at least half of each quarter spent in the office or in the field with customers, prospects, and partners, depending on the role.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
