Platform Reliability (Horizon AI)

STN • San Francisco Bay Area, CA

2d

About The Position

Horizon AI is hiring a Staff-level Platform Reliability engineer (MTS 4) to make its software platform dependable and scalable, working with the Principal Engineer. This is reliability engineering for Horizon AI’s software rather than its GPU fleet, although the two intersect when agents are trusted to take fleet actions. The role reports to the Chief AI Officer initially and later to the Senior Engineering Manager.

Requirements

• Production experience making distributed systems reliable, including caching, queuing, circuit breaking, backpressure, timeouts, and idempotency.
• Comfort with Python and containerized environments, with real exposure to Kubernetes or equivalent orchestration.
• Experience with event streaming and the failure modes of different options.
• Depth in observability, including metrics, distributed tracing, and judgment about alerting on symptoms rather than causes.
• A record of shipping reliability improvements with measurable before-and-after results.

Nice To Haves

• Experience with durable workflow orchestration.
• Experience with blue/green and canary deployment at scale.
• Experience profiling Python service performance.

Responsibilities

• Build a caching layer to prevent every agent tool call from directly hitting external systems.
• Implement event streaming to reduce service dependence on synchronous HTTP chains.
• Harden agent dispatch through health checks, circuit breakers, retry semantics, and observability that distinguishes failures from degradation.
• Build platform observability, including service-level metrics, cross-service tracing, and actionable alerting.
• Contribute to the orchestration migration and infrastructure-as-code.
• Improve deployment practices through canary deployments, traffic shifting, and reliable rollback.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed