Senior Software Engineer, AI Benchmarking
About The Position
SecureBio is looking for a Senior Software Engineer to build, scale, and run biosecurity evaluations on frontier AI systems. The tools you build will directly inform decisions made by frontier AI companies, policymakers, and researchers to improve the safety of AI systems. Our biosecurity evaluations are regularly used to audit state-of-the-art models for biological capabilities and dual-use biology risks prior to their public release. Our work has been publicly cited in model cards from OpenAI, Anthropic, and Google, and has informed safeguards on frontier models. We are a member of the NIST US AI Safety Consortium, and a partner to the European AI Office.
Requirements
- A proven track record in software engineering, including 5+ years of experience in industry roles and/or open-source contributions.
- Fluency in Python.
- Proficiency with Docker and AWS (particularly ECS).
- Experience building software tools that use or evaluate LLMs or AI agents (for example, code that orchestrates agents for a specific application, or that evaluates model capabilities on a specific task).
- Self-directed and comfortable working on small, fast-moving teams.
- A commitment to good coding practices, including knowing when and how to use (or not use) coding agents.
- Care about code quality, readability, and maintaining human oversight of human- and AI-written code.
- Alignment with SecureBio's mission of preventing catastrophic pandemics.
Nice To Haves
- Experience with AI evaluations (agent systems, scorer and grader design, agent trajectory analysis, jailbreaking or red-teaming), whether for security, safety, or model performance applications.
- Experience with Terraform or other Infrastructure-As-Code tools (e.g. Tofu or AWS CDK).
- Experience with modern, component-based web frontends (e.g. React with TypeScript).
- Experience with our specific tech stack: Evaluation frameworks: Inspect AI, LLM APIs: OpenAI-compatible, Anthropic, Together, etc., Agent frameworks: Codex, Claude Code, etc.
Responsibilities
- Perform pre-release and post-release assessments of frontier and open-source AI models using SecureBio's suite of biosecurity evaluations, which surface and measure dual-use biology capabilities in AI systems.
- Build adapters for models, identify and troubleshoot unexpected agent behaviors, and ensure that our evaluations are methodologically rigorous.
- Build, scale, maintain, and continuously improve SecureBio's suite of biosecurity evaluations.
- Improve our evaluations to use state-of-the-art agent frameworks, jailbreaking methods, and elicitation strategies.
- Develop and improve internal tools that enable our researchers to run capability assessments at scale.
- Contribute to and maintain cloud infrastructure to run our evaluations and analyses.
- Contribute analyses to our public dashboard of trends in AI biology capabilities (https://securebio.org/benchmarks/).
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
