Scientific Lead, Embedded Assessments
About The Position
Transluce is a nonprofit research lab building public infrastructure for AI evaluation and oversight. Its work with frontier AI developers, governments, and civil society organizations evaluates AI system behavior and informs policy. The organization is developing embedded evaluations in which independent teams assess frontier AI labs’ safety practices with employee-level access. The Scientific Lead for Embedded Assessments scopes, designs, and leads assessments conducted by a team of 3–5 evaluators embedded at a frontier AI lab. Engagements are expected to last 1–3 months, with the lead working on-site at the host lab to assess the coverage of its internal safety stack. The work includes identifying important open questions about model safety, auditing existing evaluations for coverage gaps, and designing experiments to investigate unexamined risks and failure modes. The lead owns the scientific content of each engagement’s public report. Between engagements, the lead runs practice evaluations on public models, trains other scientific staff, and collaborates with the engineering team to improve tooling. The role requires in-person work because embedded assessments depend in part on working physically within the host company’s facilities. Transluce is based in San Francisco and is open to sponsoring international visas.
Nice To Haves
- Excellent track record of scientific research, ideally in an area adjacent to AI alignment.
- Experience building AI evaluations, such as benchmarks, evaluation suites, or red-teaming work.
- Experience leading or managing researchers, ideally small, talented teams in a scrappy environment.
- Demonstrated ability to produce high-quality results while prioritizing under uncertainty and tight deadlines.
- Strong technical writing and ability to communicate results to less technical team members and the general public.
- Ability to quickly incorporate technical context, including code or other technical artifacts, and make progress toward a goal.
Responsibilities
- Refine the scientific scope of an assessment based on information gained while embedded.
- Review experiments and evaluations already conducted within the host organization to audit coverage.
- Design and help execute experiments to evaluate the host organization’s safety practices.
- Lead the scientific input into the final report for each embedded assessment.
- Lead assessments conducted by a team of 3–5 evaluators embedded at a frontier AI lab.
- Between engagements, lead practice evaluations on public models, train other scientific staff, and work with the engineering team to improve tooling.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
