Senior Staff Software Engineer, Agentic Systems - Moveworks

ServiceNow•Mountain View, CA

About The Position

Build and own the runtime infrastructure that powers Moveworks’ AI agents, which orchestrate, execute, and deliver agent responses to enterprise users in real time. This is a distributed systems engineering role, not an ML role. The systems support agents that plan and execute multi-step workflows, call tools, wait for human input, and resume while maintaining correctness, observability, and low latency. The role includes developing an agent orchestration engine for long-running sessions; distributed session management with lease-based ownership and crash recovery; event-driven messaging and real-time streaming; structured concurrency for concurrent session tasks; observability infrastructure; and caching and state layers.

Requirements

At least 10 years building production backend or infrastructure systems.
Deep experience in at least three of the following areas: distributed systems; concurrent or asynchronous programming; event-driven architectures; database systems for infrastructure; observability; gRPC/protobuf.
Strong proficiency in Python or Go; ideally both.
Experience designing and operating systems that handle real traffic at scale.
Comfort working with ambiguity on novel problems without textbook solutions.

Nice To Haves

Experience with distributed systems consistency models, idempotency, exactly-once delivery, and distributed locking or leasing.
Experience with Python asyncio, Go goroutines, structured concurrency, and cancellation handling.
Experience with SQS, Kafka, pub/sub, backpressure, and delivery guarantees.
Experience with DynamoDB conditional writes and transactions, and Redis connection pooling or pub/sub.
Experience with OpenTelemetry, distributed tracing, span context propagation, and Prometheus metrics.
Experience with gRPC/protobuf streaming RPCs, service interface design, and error handling patterns.

Responsibilities

Build and own runtime infrastructure for AI agents.
Develop an orchestration state machine for long-running agent sessions, coordinating planning, execution, and user interaction across LLM calls and tool invocations.
Implement distributed session management using lease-based ownership, heartbeat protocols, and checkpoint-based crash recovery.
Build event-driven message pipelines using SQS FIFO queues, Kafka consumers, gRPC, and Socket.IO.
Implement structured concurrency for concurrent session tasks, with fail-fast behavior and graceful cancellation.
Build observability infrastructure, including OpenTelemetry instrumentation and distributed trace context propagation across asynchronous boundaries.
Develop caching and state layers using Redis and DynamoDB KV stores, including per-organization and per-bot scoping, batch read optimization, and hot-reload configuration.