Agent Felet & Multi Agent Systems (Horizon AI)

STN • San Francisco Bay Area, CA

3d

About The Position

Horizon AI is building an operating system for how organizations run themselves, beginning with autonomous data centers. A fleet of AI agents senses and reasons about STN’s GPU infrastructure and increasingly takes action. The system is already in production against real fleet data: agents act where risk is bounded and outcomes are verifiable, while riskier decisions are presented to humans as recommendations. The work aims to move decisions from recommendation to action one class at a time, based on evidence. Data centers are the proving ground for technology intended to generalize beyond infrastructure.

The role owns the reasoning layer: a fleet of agents that diagnoses operational problems, decides what should happen, and explains why. It also owns the development of a way to demonstrate that agents are reliable enough to be trusted with greater autonomy. Current challenges include the absence of agent-behavior evaluation and autonomy SLOs, fragile multi-agent orchestration, and an incomplete actuation ladder for higher-risk remediations. The role works with the Chief Architect to define permitted actions and establish evidence that agents should be allowed to take them.

Requirements

• Have built and shipped LLM-based agentic systems in production, not just prototypes, including experience with failure modes such as malformed tool output, loops, stuck states, and context exhaustion.
• Have strong, specific opinions about evaluation; this is a particularly important area for the role.
• Have experience with tool and function calling at scale, LLM tracing, and online evaluation.
• Have strong systems engineering fundamentals in Python, recognizing that agents touching infrastructure are distributed systems.
• Have experience with multi-agent orchestration, handoff, and context management.

Nice To Haves

• Operational or SRE background.
• Experience with human-in-the-loop system design.
• Experience with retrieval over operational data.
• Experience defining what an autonomous system is allowed to do.

Responsibilities

• Build an agent-behavior evaluation harness with scenario suites drawn from real incidents, regression gating in CI, and metrics that detect confidently incorrect behavior.
• Define the autonomy SLO with the Chief Architect and own the evidence presented when proposing to move a decision from recommendation to action.
• Own the agent architecture, including orchestration, tool dispatch, task decomposition, escalation, and governance boundaries separating read from write actions.
• Improve how agents explain their reasoning so recommendations can be understood and acted upon.
• Continue encoding operational runbooks into agent-executable reasoning.
• Work with the Chief Architect to design higher-risk remediation actions and establish what agents are permitted to do.

What This Job Offers

• Job Type: Full-time
• Career Level: Principal
• Education Level: No Education Listed