Model Behavior Engineer, Reasoning and Decision Making

Elicit • Oakland, CA

3h • $250,000 - $320,000

About The Position

Elicit is hiring a Model Behavior Engineer to improve how AI systems support high-stakes decisions under uncertainty, initially in scientific R&D and commercial strategy. The role focuses on understanding where systems fall short, developing evaluations for complex reasoning, and building improvements that help users make better decisions. It combines careful reasoning, domain investigation, experimental design, and software engineering.

The engineer will join a cross-functional engineering team and report to an Engineering Manager, collaborating with engineering, product, design, and domain experts. The work includes designing agent environments and methods for representing problems, gathering evidence, using tools, retaining context, revising conclusions, and coordinating with other agents. It also involves making engineering decisions that support inspection, testing, reliability, and continued improvement. Initial work will mostly use existing frontier models; training or finetuning may be used when it helps users reason better.

The role includes working through real decision cases with users and experts; defining and measuring good reasoning through expert judgment, example inspection, and quantitative analysis; investigating model failures and testing behavioral improvements; designing maintainable software abstractions and evaluation systems; and owning the quality of shipped work through implementation, reviews, testing, monitoring, and integration into the product. The position may be performed remotely or from Elicit's Oakland office, with travel for quarterly retreats and coworking events.

Requirements

• Rigorous thinking about ambiguous problems, including making vague questions precise, identifying assumptions, weighing conflicting evidence, and explaining what would change one's mind.
• Curiosity across domains and ability to learn from specialists and communicate reasoning clearly to people with different expertise.
• Good experimental instincts, including devising tests without established benchmarks, distinguishing evidence from noise, recognizing when an evaluation rewards the wrong behavior, and inspecting failures in detail.
• Ability to independently write and debug code, work with data and model APIs, and take experiments into a maintained codebase.
• Comfort with Python or a comparable programming language and willingness to work across relevant parts of the technology stack.
• Practical familiarity with language models, including informed understanding of their capabilities and limitations and ability to choose and update approaches as models change.
• Strong engineering and architectural judgment, including designing coherent systems, explaining design choices, and distinguishing useful abstractions from unnecessary complexity.
• Experience building software that others can understand, use, and extend; experience may come from products, research systems, or other substantial technical projects.
• Effective use of coding assistants while taking responsibility for understanding designs and verifying results.

Nice To Haves

• Experience leveraging domain expertise across multiple scientific domains or business specialities.
• Background such as original research in economics or developing quantitative trading approaches under adversarial conditions.
• Substantial projects, thoughtful experiments, and clear explanations of technical decisions that demonstrate relevant abilities.
• Experience at senior or staff/principal level; the company is optimizing for a hire who can contribute at L4/senior level or above.

Responsibilities

• Understand real decisions with users and domain experts, including objectives, constraints, available evidence, and important unknowns.
• Define and measure good reasoning by building evaluations for underspecified problems using example inspection, expert judgment, and quantitative analysis.
• Investigate failures and test whether evaluation improvements reflect meaningful gains that generalize to new cases.
• Improve system behavior by experimenting with problem decomposition, context, tools, and ways agents seek information or check their work.
• Design agent environments and representations for problems, evidence, assumptions, and unresolved questions.
• Design software abstractions, data structures, and component boundaries for reasoning workflows and evaluations, balancing simplicity, flexibility, reliability, latency, and cost.
• Write maintainable code, participate in design and code reviews, and test and monitor changes.
• Work with engineers to integrate systems into the product and improve them based on real-world use.
• Collaborate with product, design, engineering, users, and domain experts from investigation and experimental design through implementation and product use.

Benefits

• Fully covered health, dental, vision, and life insurance for the employee, with generous family coverage and FSA/HSA.
• Flexible vacation policy, with a recommended minimum of 20 days per year, plus company holidays.
• $200 monthly wellbeing stipend.
• 401(k) with a 6% employer match.
• A new Mac and $1,000 workstation or home-office setup budget in the first year, then $500 each year thereafter.
• $1,000 quarterly AI Experimentation & Learning budget for AI tools, courses, educational resources, or AI-focused conferences and events.
• Team administrative assistant support for personal and work tasks.
• Above-market equity for all roles and employee-friendly equity terms.
• Flexible work environment, with the option to work from the Oakland office or remotely, subject to travel for retreats and coworking events.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed