Senior / Staff AI Engineer — Evals & Model Quality

FurtherAI • San Francisco, CA

2d • $300,000 - $350,000 • Onsite

About The Position

FurtherAI is hiring a Senior / Staff AI Engineer to own how the company evaluates and continuously improves AI-agent quality across its platform. The role focuses on building evaluation frameworks and infrastructure to measure agent accuracy, quality, and reliability across workflows, Assistant, document processing, and computer use. The engineer will build real-world insurance datasets and benchmarks, evaluate complex multi-step agent behavior, develop automated grading systems, and embed evaluation into development and release processes. The role also includes benchmarking frontier and open-weight models and turning production failures, customer feedback, and agent traces into measurable improvements.

FurtherAI builds AI agents for the insurance industry. The company describes itself as deeply technical, with engineers having significant input on product direction, and offers an in-office culture in San Francisco five days per week.

Requirements

• At least 5 years of software engineering experience.
• Strong hands-on experience building production AI/ML systems.
• Experience building evaluation frameworks, LLM-based applications, model benchmarking systems, or applied AI infrastructure.
• Strong understanding of modern LLMs, agent architectures, evaluation methodologies, and model performance tradeoffs.
• Strong Python engineering skills and the ability to build reliable, production-grade systems.
• Ability to independently own ambiguous technical problems from research and experimentation through production.
• Clear communication skills, both in person and in writing.
• Drive, grit, and ownership.

Nice To Haves

• Experience with LLM-as-judge, human evaluation systems, agent trajectory evaluations, or large-scale benchmarking.
• Experience with open-weight models, fine-tuning, post-training, or model routing.
• Background in B2B SaaS and 0-to-1 experience.

Responsibilities

• Build evaluation frameworks and infrastructure to measure the accuracy, quality, and reliability of AI agents across workflows, Assistant, document processing, and computer use.
• Develop golden datasets, benchmarks, and regression suites in collaboration with Forward Deployed Engineers and domain experts, capturing real-world insurance tasks and edge cases.
• Design evaluations for multi-step agent trajectories, tool use, reasoning, and end-to-end task completion.
• Develop LLM-as-judge evaluations, deterministic graders, human evaluation workflows, and rubrics that reflect real-world quality.
• Integrate offline and online evaluations into development and release processes so engineers can measure improvements, catch regressions, and ship confidently.
• Benchmark frontier and open-weight models across accuracy, latency, cost, and reliability, and use the results to inform model selection, upgrades, and production releases.
• Turn production failures, customer feedback, and agent traces into new evaluation cases and measurable improvements in agent quality.
• Build end-to-end evaluation frameworks, reusable libraries of golden datasets and benchmarks, automated model benchmarking systems, and production evaluation pipelines.

Benefits

• Fully covered, best-in-class health, dental, and vision benefits.
• Competitive compensation.
• Meaningful stock options.
• Company 401(k).
• Unlimited PTO.
• Outstanding in-office culture in the heart of San Francisco, five days per week.
• Lunch and dinner onsite.
• Team events, such as happy hours and off-sites.
• Pre-tax commuter benefits.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed