Senior Software Engineer, Agent Oversight

Scale AI • San Francisco, CA

2d • $216,000 - $270,000

About The Position

Scale’s Applied Intelligence Systems team, part of the Scale Generative AI Platform, builds infrastructure and tooling for production agentic AI across enterprise and government use cases. The team combines platform engineering with applied ML research, design, and evaluation to make agent systems reliable at scale.

As a Software Engineer on Agent Oversight, you will build platform infrastructure that enables production agents to be observed, evaluated, and improved at scale. You will develop observability tooling, evaluation harnesses, and pipelines connecting these capabilities to improvement loops. You will own systems end-to-end while maintaining rigorous technical standards, and partner with ML engineers on production workflows where platform engineering intersects with evaluation or improvement systems.

The role includes building features from system design through implementation, debugging, and testing; owning reliability, scalability, and observability; and working with product, forward deployed engineering, and customers to translate deployment requirements into platform features. You will also participate in high-velocity experimentation to validate platform capabilities against real customer usage.

Requirements

• At least 4 years of professional software engineering experience, with strong fundamentals in backend and distributed systems, APIs, and data pipeline design.
• Hands-on experience building production software for ML/LLM-powered products or platforms, such as evaluation systems, observability or monitoring, experimentation infrastructure, agent runtimes, model-serving-adjacent services, or telemetry and data pipelines.
• Working knowledge of how LLM or ML systems behave in production, including evaluation signals, failure modes, prompt and tool-calling workflows, experiment results, data quality issues, and tradeoffs between offline evaluations and live customer behavior.
• Experience partnering closely with ML engineers or applied researchers to turn prototypes, evaluation loops, or model-improvement workflows into reliable platform capabilities; ownership of model training, modeling strategy, or research direction is not required.
• Experience building infrastructure or platforms that other engineering teams build on, such as internal platforms or developer tools.
• Track record of owning features or components end-to-end, from design through production, within a larger platform or system.
• Comfort operating in an ambiguous, fast-changing domain where tooling and best practices are still being defined.
• Strong problem-solving skills and ability to work independently or as part of a close-knit, cross-functional team.
• Excitement about working directly with ML engineers and customer-facing teams, including challenging assumptions in designs and metrics where platform behavior, model behavior, and customer needs intersect.
• Ability to give direct, substantive feedback on designs and code, receive feedback constructively, and mentor others as they grow.

Nice To Haves

• Deep experience building or maintaining observability, monitoring, or evaluation systems for ML/LLM-powered products in production.
• Familiarity with agent architectures, including tool use, planning, and multi-agent orchestration.
• Exposure to MLOps, feature stores, model serving, or experiment infrastructure.
• Experience working in regulated or enterprise contexts.
• Experience reviewing others’ technical designs or mentoring engineers at a senior or staff level.

Responsibilities

• Design and build core platform capabilities for deploying, monitoring, and evaluating agentic applications in production.
• Build reliable APIs and data pipelines to capture agent telemetry, evaluation signals, and performance metrics at scale.
• Partner with ML engineers on evaluation and improvement workflows, bringing sufficient ML fluency to reason about model behavior, evaluation quality, and improvement loops while owning the supporting software systems.
• Own the reliability, scalability, and observability of platform components serving multiple concurrent enterprise and government customers.
• Work cross-functionally with product, forward deployed engineering, and customers to translate real-world deployment requirements into platform features.
• Build features end-to-end, including system design, implementation, debugging, and testing.
• Participate in high-velocity experimentation to validate platform capabilities against real customer usage.

Benefits

• Comprehensive health coverage.
• Dental coverage.
• Vision coverage.
• Retirement benefits.
• Learning and development stipend.
• Generous paid time off.
• Potential eligibility for a commuter stipend.
• Equity-based compensation for eligible roles, subject to Board of Director approval.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed