Staff Applied Scientist

Robots and Pencils

2d•$200,000 - $300,000

About The Position

Robots & Pencils is seeking a Staff Applied Scientist to lead the design and development of an evaluation framework for an enterprise planner-based agent. The role works directly with a client’s engineering team to benchmark the agent across planner models, executor LLMs, and inference optimization strategies. Proposed changes are evaluated through controlled, statistically sound experiments with measurable financial outcomes. This is a research-heavy, customer-facing role with substantial hands-on development. The scientist owns both the experimental methodology and the repeatable tooling: a scenario-driven user simulator, isolated experiment execution, statistical comparison, and an automated-insights dashboard hosted locally for the client to adopt. The work spans an evaluation pipeline, grading and statistics, and a dashboard and experiment campaign, culminating in actionable recommendations. The role establishes evidence for decisions about planner models, executors, and optimization levers, quantifies quality and cost effects at production volume, and assesses how well the planner generalizes from data analytics to other tasks. It also involves setting experimental standards, mentoring engineers, and communicating results to scientific and business audiences.

Requirements

Experience with LLM and agent evaluation methods, such as programmatic state-based verifiers, LLM-as-judge or agent-as-judge, trajectory-level scoring, and user simulation.
Experience building or evaluating planner-executor or other multi-step, tool-using agent architectures in production, not just prototypes.
Strong background in experimental design and statistical inference, including paired designs, non-inferiority testing, and power analysis, with the judgment to identify inconclusive results.
Experience with multi-objective optimization or cost-aware machine learning, including Pareto frontier analysis and converting results into projected dollar savings.
Familiarity with LLM routing, model cascades, or model selection, and understanding of LLM inference internals and cost levers such as prompt and plan caching, context compaction, reasoning budgets, and cache-aware model switching.
Experience with observability and tracing for LLM systems, including attribution of tokens, cache usage, latency, and cost per request.
Ability to build production-quality research tooling, including Python services, containerized and sandboxed execution, and data pipelines.
Prior customer-facing or consulting experience scoping and delivering research engagements.
Ability to explain technical findings clearly to non-technical audiences, translate them into business impact, and provide deeper technical detail when challenged.

Nice To Haves

Publications or open-source contributions in LLM evaluation, agents, or efficient inference.
Experience building internal dashboards or web UIs that a client team can adopt.
Hands-on experience with an LLM gateway or tracing stack, such as LangFuse or LangSmith, to attribute cost and latency at the request level.
Experience in a regulated or high-stakes enterprise domain where evaluation rigor has significant consequences.
A perspective on cost-quality tradeoffs gained from running them in production.

Responsibilities

Design the end-to-end evaluation workflow for a planner-executor agent, including scenarios, user simulation, configuration, execution, grading, comparison, and insights.
Define the scenario schema, grading specification, and validators with the client’s team so client-created scenarios can be used by the framework.
Build an LLM-based user simulator with a hidden goal and stop rule for realistic multi-turn conversations, and assess its realism against individual production logs.
Build isolated, parallel experiment execution using configuration files, one run per configuration and scenario, and a harness adapter that routes model traffic through the client’s existing LLM gateway.
Package optimization levers as plug-ins and implement a shortlist of three to four for the first version, such as cascade-on-failure, cache-aware model switching, and prompt caching.
Tag every model call at the gateway and send traces to a trace store, attributing tokens, cache usage, latency, and cost to each request.
Grade each run against its expected end state as the primary signal, with optional LLM judges for answer and plan quality.
Compute verified success, cost per success, p95 latency, and failures by pipeline stage.
Compare experiments on the same scenarios using paired methods, confidence intervals, per-scenario win/loss/tie results, and a non-inferiority test for quality impact; clearly identify inconclusive results.
Convert experiment differences into projected dollar savings at production volume using real deployment costs, and plot the cost-versus-quality frontier.
Build an automated insight engine in which a statistics layer computes results and an LLM writes them up, with each ranked finding linked to its evidence.
Build a locally hosted, API-first dashboard with Launch, Live, Results, Compare, Insights, and Traces views, and enable one-command local deployment for later client-platform integration.
Run the staged campaign: production baseline, model screen, lever screen, and confirmation of the best combination.
Analyze generalization by scenario category, optionally alongside a small data-analytics anchor set.
Deliver a final recommendation with confidence intervals and dollar impact, along with a runbook and integration guide to support go/no-go decisions.
Partner with client engineers and leadership to agree on scenarios, margins, and success criteria, and explain findings to scientists and business stakeholders.
Write technical reports and leadership summaries that translate results into business impact.
Set experimental standards for the pod and mentor engineers on rigorous agent evaluation, including AI-assisted development tools such as Claude Code.