AI Applied Scientist

Wizard

7h•$225,000 - $280,000•Remote

About The Position

Wizard is seeking an Applied Scientist to lead the measurement, understanding, and improvement of its AI agent's accuracy. This role involves applied ML, evaluation science, and product strategy. The successful candidate will define evaluation criteria, develop measurement systems, and drive scientific improvements, including fine-tuning LLM judges. This is a foundational role on the science team, with opportunities to expand into broader applied science areas as the agent's capabilities grow.

Requirements

5+ years in Applied ML, AI Research, or Applied Science (PhD or equivalent depth strongly preferred)
Hands-on experience evaluating modern AI/ML systems: LLMs, agents, ranking, or recommendations
Direct experience with LLM-based systems: judge models, RAG, prompt engineering, fine-tuning, RLHF, or similar
Strong experimentation foundations: A/B testing, causal inference, statistical rigor
Proven ability to operate in ambiguity: defining problems, not just solving pre-defined ones
Clear, structured communication that influences across ML, engineering, and product

Responsibilities

Define and evolve accuracy metrics across the full shopping experience (retrieval, ranking, recommendations, outcomes)
Design and run experiments to measure improvements and regressions
Build and maintain evaluation datasets, benchmarks, and scoring frameworks
Improve the LLM judges that power our evaluation pipeline: prompting, calibration, and fine-tuning where it matters
Translate ambiguous product questions into clear, measurable hypotheses and analysis
Partner with ML Engineers to validate model changes and guide iteration
Identify failure modes and edge cases, and drive improvements through data
Make agent performance visible, trusted, and actionable across product and engineering