Research Lead, Evaluations and Benchmarks

Alice

4d

About The Position

Alice is a trust, safety, and security company built for the AI era. We safeguard the communicative technologies people use to create, collaborate, and interact- whether with each other or with machines. Alice clients are the top 8 AI Labs in the world. In a world where AI has fundamentally changed the nature of risk, Alice provides end-end coverage across the entire AI lifecycle. We support frontier model labs, enterprises, and UGC platforms with a comprehensive suite of solutions: from model hardening evaluations and pre-deployment red-teaming to runtime guardrails and ongoing drift detection. Alice is widely considered a global leader in online safety and AI security. We have some of the most forward-thinking and passionate minds in the world working to safeguard over 3 billion users across the largest AI and tech platforms. If you're creative and driven to secure the future of AI, we want to hear from you!

Requirements

• PhD or Masters in computer science, machine learning or a related field, or equivalent depth from industry research.
• 3+ years building and running safety or security evaluations for language models in production, at an AI lab, a model provider, or a safety and security research organization.
• 5+ relevant research publications in the field of AI safety and security including lead author on at least 2 of them.
• Strong engineering skills: Evaluation harnesses, distributed inference, vLLM, reading and fixing codebases.
• Ability to build a taxonomy, not just score against one.
• Ability to direct a researcher and two freelancers without formal management.
• Strong English, written and spoken, for internal communication across time zones.
• Curiosity about the harms themselves, with a willingness to learn a new subject every three weeks.

Nice To Haves

• Post-training experience: SFT, DPO, GRPO. Reward design for subjective and safety-relevant targets.
• Agentic evaluation experience: tool use, orchestration, permissions, prompt injection.
• Publications at top conferences.
• Willingness to present your own work on a client call.
• Strong communication - both verbal and written, ability to present to large and/or senior audiences.
• Travel to conferences at least 3 times a year.

Responsibilities

• Ship a benchmark every two to three weeks, with size following the subject. A chat-based taxonomy can carry 100 evals. An agentic or GRPO benchmark is closer to 20, because each one is expensive to read. Sensitivity decides what ships publicly and what goes to the labs alone.
• Own the quality bar: Ensure a frontier lab can rerun the set and get the same numbers, verifiers hold, rubrics are clear, distribution is sane, and subject matter experts find the taxonomy novel. This involves reading evals yourself, screening with a model, and pushing back on researchers when items do not match the taxonomy.
• Run the process: Hold the plan and calendar, keeping other researchers on timeline. Direct two or three freelancers (SMEs) yourself ad-hoc when needed.
• Set the roadmap with the forum: Monthly meetings with the CTO and research leads to discuss inputs from research teams, client requests, and news to create a revised quarterly release plan tied to target accounts.
• Stay ahead of the curve: Dedicate approximately 20% of time to the ecosystem, reading research, maintaining contacts within labs, understanding their challenges, and attending a couple of conferences annually. Engage in weekly conversations with individuals from AI labs.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior