Senior Staff Software Engineer, Agentic Systems - Moveworks

ServiceNow•Mountain View, CA

2d•$201,300 - $352,300

About The Position

Moveworks is building runtime infrastructure that powers AI agents, enabling them to orchestrate and execute workflows and deliver responses to enterprise users in real time. This is a distributed systems engineering role, not a machine learning role. The systems support agents that plan, execute multi-step workflows, call tools, wait for human input, and resume while maintaining correctness, observability, and low latency. The engineer will build and own the agent orchestration engine, distributed session management, event-driven messaging, structured concurrency, observability infrastructure, and caching and state layers. The role calls for deep experience in at least three specified areas of distributed systems and infrastructure, along with 10 or more years building production backend or infrastructure systems.

Requirements

At least 10 years of experience building production backend or infrastructure systems.
Deep experience in at least three of the following areas: distributed systems; concurrent or asynchronous programming; event-driven architectures; database systems for infrastructure; observability; gRPC/protobuf.
Strong programming skills in Python or Go; ideally both.
Experience designing and operating systems that handle real traffic at scale.
Comfort working with ambiguity and novel problems without textbook solutions.

Nice To Haves

Experience with distributed systems concepts such as consistency models, idempotency, exactly-once delivery, and distributed locking or leasing.
Experience with Python asyncio, Go goroutines, structured concurrency, and cancellation handling.
Experience with message queues such as SQS and Kafka, pub/sub, backpressure, and delivery guarantees.
Experience with DynamoDB conditional writes and transactions, and Redis connection pooling or pub/sub.
Experience with OpenTelemetry, distributed tracing, span context propagation, and Prometheus metrics.
Experience with gRPC/protobuf streaming RPCs, service interface design, and error-handling patterns.

Responsibilities

Build and own an agent orchestration engine that manages long-running agent sessions across planning, execution, user interaction, LLM calls, and tool invocations.
Develop distributed session management using lease-based ownership, conditional writes, heartbeat protocols, and checkpoint-based crash recovery.
Build event-driven messaging pipelines using SQS FIFO queues, Kafka consumers, and real-time streaming with gRPC and Socket.IO.
Implement structured concurrency with Python asyncio TaskGroups for concurrent per-session tasks, including message polling, lease heartbeats, output publishing, and orchestrator execution; support fail-fast behavior and graceful cancellation.
Develop observability infrastructure with OpenTelemetry, distributed trace context propagation across asynchronous boundaries, and custom span lifecycle management for long-running sessions.
Build caching and state layers using Redis and DynamoDB KV stores, including per-organization and per-bot scoping, batch read optimization, and hot-reload configuration.
Design and operate production systems that handle real traffic at scale.