Sr Software Development Engineer, Business Process Optimization

Workday•Pleasanton, CA

1d•$148,200 - $264,000•Hybrid

About The Position

The Senior Software Development Engineer on Workday’s Business Process Optimize (BPO) team will drive the technical design and execution of core components for a next-generation Agentic Workflow and Analytics Platform. The team builds platform infrastructure for scalable, high-availability data ingestion and event-driven systems supporting enterprise billing integrity and administrator workflows. The platform serves more than 100 active Tier-0 enterprise customers and millions of daily API requests. This hands-on role involves solving multi-petabyte-scale data challenges and building low-latency infrastructure using ClickHouse and big-data streaming technologies such as Kafka, Spark/Flink, Iceberg, and S3. The engineer will expand distributed tracing and telemetry to cover LLM and agentic workflows, providing visibility into agent execution, tool calls, and prompt/response chains in global, multi-tenant environments. The position collaborates with cross-functional partners to deliver releases and improve platform reliability, performance, and observability.

Requirements

At least 8 years of software development engineering experience architecting, building, and scaling robust and efficient software systems.
At least 5 years of experience designing, building, and operating complex distributed system architectures, with demonstrated high availability (99.9% uptime) and fault tolerance.
At least 5 years of experience writing production-grade code in at least two of Java, Go, Scala, or Python.
Strong ability in data partitioning, dynamic sharding, high-dimensional vector retrieval, and caching strategies for high-throughput indexing and sub-second queries over massive datasets.
Experience with gRPC, REST, and OTLP for scalable distributed APIs, along with object-oriented design skills, clean abstractions, domain-driven design, and microservices patterns.
Knowledge of distributed load and integration testing, telemetry-driven regression checks, and CI/CD automation.
Experience with Git, GitHub, or Bitbucket, including advanced branching, merging, and pull request practices.
Proficiency in at least one of Java, Go, Scala, or Python, and knowledge of enterprise security practices such as mTLS, RBAC/ABAC, and encryption at rest and in transit.
Ability to mentor engineers and technically lead cross-functional pods, with strong technical writing skills for architecture specifications, API contracts, runbooks, and LLM/agentic tracing documentation.
Bachelor’s degree in Computer Science, Software Engineering, or a related field, or equivalent experience.

Nice To Haves

Master’s degree in Computer Science or Distributed Systems.
Experience with ClickHouse, Kafka, Spark/Flink, Apache Iceberg, S3, Grafana Tempo, LangSmith, or LangChain.

Responsibilities

Design, implement, and maintain scalable, fault-tolerant microservices and high-throughput data ingestion pipelines capable of sub-second query execution.
Extend distributed tracing through OpenTelemetry (OTLP) to monitor agent execution paths, tool invocations, and LLM reasoning steps across enterprise workloads.
Optimize query performance and data retention strategies on ClickHouse, Grafana Tempo, and Apache Iceberg to support platform availability.
Build high-performance gRPC and REST APIs for integration with the BPO Model Context Protocol (MCP) framework and Agent Ready Tools.
Collaborate with Product, Data Engineering, SRE, Legal/Compliance, QA automation leads, and cross-team partners to deliver feature releases on tight timelines.
Author technical design specifications, runbooks, and system architecture documentation.
Contribute to CI/CD pipelines, load testing, and multi-region failover strategies.
Mentor engineers and technically lead cross-functional pods.