Research Lead, Evaluations and Benchmarks

Alice • New York, NY

3d

About The Position

Alice is a trust, safety, and security company built for the AI era. We safeguard the communicative technologies people use to create, collaborate, and interact—whether with each other or with machines. Alice clients are the top 8 AI Labs in the world. In a world where AI has fundamentally changed the nature of risk, Alice provides end-to-end coverage across the entire AI lifecycle. We support frontier model labs, enterprises, and UGC platforms with a comprehensive suite of solutions: from model hardening evaluations and pre-deployment red-teaming to runtime guardrails and ongoing drift detection. Alice is widely considered a global leader in online safety and AI security. We have some of the most forward-thinking and passionate minds in the world working to safeguard over 3 billion users across the largest AI and tech platforms. If you're creative and driven to secure the future of AI, we want to hear from you!

Requirements

• PhD or Masters in computer science, machine learning or a related field, or equivalent depth from industry research.
• 3+ years building and running safety or security evaluations for language models in production, at an AI lab, a model provider, or a safety and security research organization.
• 5+ relevant research publications in the field of AI safety and security, including lead author on at least 2 of them.
• Strong engineering skills, including evaluation harnesses, distributed inference, vLLM, and the ability to read and fix codebases.
• Ability to build a taxonomy, not just score against one.
• Ability to direct a researcher and two freelancers without managing them formally.
• Strong English, written and spoken, for internal communication across time zones.
• Curiosity about the harms themselves, with a willingness to learn a new subject every three weeks.

Nice To Haves

• Post-training experience: SFT, DPO, GRPO. Reward design for subjective and safety-relevant targets.
• Agentic evaluation experience: tool use, orchestration, permissions, prompt injection.
• Publications at top conferences.
• Willingness to present your own work on a client call.
• Strong communication skills, both verbal and written, with the ability to present to large and/or senior audiences.
• Travel to conferences at least 3 times a year.

Responsibilities

• Ship a benchmark every two to three weeks, with size following the subject. A chat-based taxonomy can carry 100 evals. An agentic or GRPO benchmark is closer to 20, because each one is expensive to read.
• Own the quality bar, ensuring that frontier labs can rerun sets and get the same numbers, verifiers hold, rubrics are clear, distribution is sane, and subject matter experts find the taxonomy novel. This includes reading evals yourself, screening with a model, and identifying items that do not match the taxonomy.
• Run the process by holding the plan and calendar, keeping other researchers on timeline, and directing two or three freelancers (SMEs) ad-hoc when needed.
• Set the roadmap with the forum, meeting roughly monthly with the CTO and pod/research leads to discuss inputs from research teams, client requests, and news, to produce a revised release plan for the quarter tied to target accounts.
• Dedicate around 20% of time to the ecosystem by reading research, maintaining contacts within labs, understanding their challenges, and traveling to a couple of conferences a year. Engage in weekly conversations with individuals from the labs.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior