Member of Technical Staff (Software Engineer, Enterprise Agent Evals & Efficiency)

Perplexity • Seattle, WA

4h • $220,000 - $405,000

About The Position

The Enterprise Agent Evals & Efficiency team improves Perplexity's agents’ ability to complete real work across enterprise connectors, tools, and data. The team defines successful outcomes, builds evaluations to measure them, identifies agent failure points, and ships changes that improve reliability while reducing time and compute.

This hands-on engineering role has end-to-end ownership, from building realistic enterprise evaluation environments to shipping improvements across the agent harness, tools, and skills. The engineer creates complex scenarios spanning connected systems, data, and permissions; uses them to identify agent shortcomings; and improves tools, skills, context, and models. In collaboration with AI Research, Connectors, and Platform teams, the role turns experimental results into measurable gains in task success, latency, and cost per successful task.

Requirements

• At least 5 years of software engineering or machine learning engineering experience, with a track record of owning complex production systems from design through deployment and operation.
• Strong system design skills, including experience building reliable, scalable infrastructure for experimentation, evaluation, or data-intensive applications.
• Strong proficiency in a backend language such as Python, Go, or Rust, and ability to work effectively across a multi-language production stack.
• Hands-on experience building, evaluating, or improving LLM-based applications or agents, including tool use and multi-step workflows.
• Deep expertise in at least one of the following: evaluation infrastructure, benchmark design, and automated grading; agent harnesses, tool interfaces, and reusable skills for complex workflows; realistic test environments or simulations spanning multiple systems, datasets, permissions, and stateful workflows; or model selection, routing, and inference efficiency, including context management and caching.
• Strong experimental and statistical judgment, including selecting representative tasks, defining meaningful success criteria, diagnosing failures, and distinguishing real improvements from noise.
• Sound technical judgment in balancing task quality, latency, reliability, and cost, with a track record of translating evaluation findings into shipped improvements.
• Track record of leading ambiguous, cross-team initiatives from architecture through production adoption, setting technical direction, and raising engineering quality through mentorship.

Responsibilities

• Build realistic enterprise evaluation environments spanning connected applications, representative datasets, user roles, permissions, and complex multi-step workflows with reproducible system state.
• Own evaluation infrastructure, task suites, and automated graders that measure agent performance on enterprise work, calibrated against human judgment and grounded in end-to-end outcomes.
• Analyze agent trajectories and production failures, identify root causes, and ship improvements to the agent harness, tools, and skills that measurably improve task success and reliability.
• Develop reusable skills and improve tool descriptions, schemas, and result handling so agents select appropriate tools, execute workflows correctly, and recover from errors with fewer unnecessary calls.
• Build and evaluate model-selection and routing strategies that match tasks to appropriate models and effort levels while balancing quality, latency, reliability, and cost.
• Improve agent efficiency through context management, skill loading, caching, and execution strategies that reduce wasted tokens and cost per successful task without compromising quality or safety.
• Build regression gates and launch-readiness signals for model, connector, tool, skill, and harness changes, and validate that evaluation gains translate into better production outcomes.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed