Sr. Software Development Engineer

Workday • Vancouver, BC

3d • $148,200 - $264,000 • Hybrid

About The Position

The Senior Software Development Engineer on Workday’s AI Model Serving team is a technical leader who helps shape the platform’s vision and direction alongside the engineering manager. The role makes critical design decisions, drives team outcomes, and helps foster a positive and inclusive culture. The engineer builds and operates infrastructure that serves production AI workloads, including traditional machine-learning models and large language models used by Workday agents.

The team operates services that connect to vendor-hosted LLMs and host Workday’s internal models. It serves thousands of traditional ML models across sharded Ray Serve clusters, maintains Workday’s production model registry, and handles approximately 2,000 requests per second, with peaks above 10,000 requests per second in its largest clusters. Planned work includes supporting additional production AI agents, hosting and tuning open-weight LLMs, improving latency and reliability, and strengthening vendor integrations and cost governance.

Requirements

• At least 6 years of related software development experience, focused on building and operating large-scale distributed systems.
• Bachelor’s degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience.
• Deep experience designing, building, and scaling production-grade distributed systems.
• Knowledge of the full software development lifecycle, including coding standards, testing, code reviews, source control, and deployment.
• Deep proficiency in Python, including production-level coding and building systems with Python-based frameworks.
• Hands-on experience deploying and scaling workloads on Kubernetes, particularly GPU resource management.
• Ability to optimize GPU utilization for hosting and tuning smaller open-weight LLMs with modern inference engines such as vLLM or TGI.
• Understanding of GPU memory constraints, serving tuned models such as LoRA, and autoscaling based on hardware metrics.
• Familiarity with large language models and traditional machine-learning models, including serving, scaling, and production monitoring.
• Ability to design abstractions that effectively serve both LLMs and traditional ML models.
• Ability to design and maintain monitoring strategies that provide insight into system health, performance, and cost.
• Excellent written and verbal communication skills, including writing clear design documents, explaining complex technical ideas, and building consensus across teams.
• Collaborative engineering approach and experience mentoring other engineers.

Responsibilities

• Lead technical work and make design decisions that improve platform performance, reliability, and scalability.
• Design, implement, and maintain large-scale systems for moving machine-learning models into production.
• Write design documents to build consensus for new system components and enhancements to existing components.
• Evaluate and adopt technologies available within Workday and across the broader industry.
• Troubleshoot, improve, and scale continuous-integration software pipelines.
• Build relationships with software engineers, machine-learning engineers, and data scientists on partner teams.
• Respond to alerts and debug production issues to maintain platform health and reliability.
• Review pull requests and promote consistency, performance, readability, and security across codebases.
• Develop documentation to share knowledge with other engineers.
• Mentor engineers and foster a collaborative, inclusive team environment.

Benefits

• The role may be eligible for the Workday Bonus Plan or a role-specific commission or bonus.
• Annual refresh stock grants may be included in the total compensation package.
• Workday offers a flexible work approach combining in-person and remote work, with at least half of each quarter spent in the office or in the field, depending on role.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior