Sr Software Development Engineer, Business Process Optimization

Workday • Pleasanton, CA

1d • Hybrid

About The Position

The Senior Software Development Engineer on Workday’s Business Process Optimize (BPO) team will drive technical design and execution for core components of a next-generation Agentic Workflow and Analytics Platform. The role focuses on AI-driven automation, Model Context Protocol (MCP), and enterprise-scale observability, serving more than 100 active Tier-0 enterprise customers and supporting millions of daily API requests.

The engineer will build and operate distributed systems, high-availability data pipelines, and event-driven data flows across global product regions. The work includes multi-petabyte-scale data challenges; low-latency infrastructure using ClickHouse and big-data streaming technologies; and extending distributed tracing and telemetry to LLM and agentic workflows using tools such as LangSmith and LangChain. The position is associated with Workday’s Flex Work approach, which combines in-person and remote work and calls for spending at least 50% of each quarter in an office or in the field with customers, prospects, and partners, depending on the role.

Requirements

• At least 8 years of software development engineering experience architecting, building, and scaling robust and efficient software systems.
• At least 5 years of experience designing, building, and operating complex distributed system architectures, with demonstrated high availability (99.9% uptime) and fault tolerance.
• At least 5 years of experience writing production-grade code in at least two of Java, Go, Scala, or Python.
• Strong capability in data partitioning, dynamic sharding, high-dimensional vector retrieval, and caching strategies for high-throughput indexing and sub-second queries over massive datasets.
• Experience with gRPC, REST, and OTLP for scalable distributed APIs, and object-oriented design, clean abstractions, domain-driven design, and microservices patterns.
• Knowledge of distributed load and integration testing, telemetry-driven regression checks, and CI/CD automation.
• Experience with Git, GitHub, or Bitbucket, including advanced branching, merging, and pull request practices.
• Proficiency in Java, Go, Scala, or Python, and knowledge of enterprise security practices such as mTLS, RBAC/ABAC, and encryption at rest and in transit.
• Ability to mentor engineers and technically lead cross-functional pods.
• Strong technical writing skills for architecture specifications, API contracts, runbooks, and LLM/agentic tracing documentation.
• Bachelor’s degree in Computer Science, Software Engineering, or a related field, or equivalent experience.

Nice To Haves

• Master’s degree in Computer Science or Distributed Systems.
• Experience with ClickHouse, Kafka, Spark or Flink, Iceberg, and S3.
• Experience with Grafana Tempo and OpenTelemetry (OTLP).
• Experience with LangSmith and LangChain.

Responsibilities

• Design, implement, and maintain scalable, fault-tolerant microservices and high-throughput data ingestion pipelines capable of sub-second query execution.
• Extend distributed tracing through OpenTelemetry (OTLP) to monitor agentic execution paths, tool invocations, and LLM reasoning steps across enterprise workloads.
• Optimize query performance and data retention strategies on ClickHouse, Grafana Tempo, and Apache Iceberg to support platform availability.
• Build high-performance gRPC and REST APIs for tool integration with BPO’s Model Context Protocol framework and Agent Ready Tools.
• Collaborate with Product, Data Engineering, SRE, Legal/Compliance, QA automation leads, and cross-team partners to deliver feature releases on tight timelines.
• Write technical design specifications, runbooks, and system architecture documentation.
• Contribute to CI/CD pipelines, load testing, and multi-region failover strategies.
• Mentor engineers and provide technical leadership to cross-functional pods.

Benefits

• The role may be eligible for the Workday Bonus Plan or a role-specific commission/bonus.
• Annual refresh stock grants may be part of the total compensation package.
• Flexible scheduling through Workday’s Flex Work approach.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior