Researcher

Theory AI•Washington, DC

6h•Remote

About The Position

This is a Remote (work from home) position. Client work surfaces the same unsolved problems repeatedly: how to evaluate refusal behaviour in a way that transfers across deployments, how to measure annotation quality when there is no gold standard, how to say something honest about a model whose failure modes are rare and consequential. This role exists to work on those problems properly rather than around them each time. Part of your output is internal: methodology the delivery teams adopt. Part of it is external, because publishing is how a firm our size builds a reputation for knowing what it is talking about. You will not be isolated from delivery. The problems come from real engagements, and the answers go back into them.

Requirements

A research background in machine learning, statistics, or a quantitative field.
Demonstrated original work, published or otherwise verifiable
Strong Python and the ability to build your own experimental tooling
Writes clearly for readers who are not specialists in your subfield

Nice To Haves

Published on evaluation, benchmarking, or measurement validity
Experience with human annotation as a research instrument
Comfortable presenting to non-research audiences, including clients

Responsibilities

Investigate evaluation and measurement problems arising from client engagements
Develop methodology the delivery teams can actually adopt, with the tooling to make it usable
Publish findings externally where the work supports it
Review the methodological soundness of what we claim to clients
Keep the firm current on relevant research and translate it into practice