Principal Research Scientist

3d•$190,000 - $235,000

About The Position

G2 is hiring its most senior applied research professional for its work developing independent, rigorous evaluations of AI agents performing realistic business tasks. The role owns the research agenda and evaluation methodology, ensuring that findings are meaningful, defensible, and kept current as models change. The work is hands-on and contributes directly to G2’s current products, including collaborating with engineers and product managers to scope features, implement evaluation methods in production, and respond to customer and business needs. The Principal Research Scientist will design realistic environments and task sets, develop reliable measurement approaches using automated checks, LLM-based judges, and human review, and validate tasks and graders by quantifying noise and determining whether performance differences are meaningful. The role also communicates and defends results to buyers, customers, partners, and the research community. The position provides technical leadership through mentoring researchers and engineers, reviewing methodology, and creating documentation and repeatable standards. It involves tracking research and benchmark developments, translating relevant ideas into product and roadmap decisions, collaborating with data and go-to-market teams, and writing methodology documentation and, where appropriate, papers and public posts. Research directions may include simulated users and environments, realistic persona simulation, judge reliability, voice and multimodal agents, long-horizon workflows, and test-set contamination prevention.

Requirements

Experience working with evaluation design or dataset curation in multi-agentic systems.
At least 5 years building or evaluating ML systems, including hands-on work with LLMs or LLM-based agents.
Graduate degree (MS or PhD) in machine learning, NLP, computer science, statistics, or a related field, or equivalent applied research experience.
Track record of original machine-learning ideas or clear improvements to existing methods, demonstrated through first-author publications or projects led end-to-end.
Ability to own a research agenda, select worthwhile problems, and carry long-running projects without direction.
Track record of shipping ML work in a product setting and working closely with engineers and product managers.
Experience setting technical direction for others, including mentoring researchers or engineers, reviewing methodology, and improving the quality of others’ work.
Familiarity with AI benchmarking and understanding of how benchmark scores can mislead.
Sound statistical judgment covering experimental design, variance, significance, and agreement between human and automated raters.
Good judgment about trade-offs, including choosing practical solutions that serve customers now over more complete solutions later.
Clear writing and ability to explain methodology to non-specialists and defend it to skeptical audiences.

Nice To Haves

At least 2 years of experience managing small to medium teams (listed as preferred).
Research experience in agent evaluation, user simulation, or human- and AI-based judging.
Experience with conversational, voice, retrieval-based, or tool-using AI systems.
Contributions to open-source evaluation tools or public leaderboards.
Familiarity with business software workflows.
A considered view on the effects of AI technology on the people and markets it reaches.
Publications at ACL, EMNLP, NeurIPS, ICML, ICLR, COLM, or similar venues.

Responsibilities

Own and maintain G2’s AI-agent evaluation methodology, defend its standards, and adapt them as models change.
Work with engineers and product managers to build and improve evaluations for current products, scope features, productionize methods, and iterate based on customer and business needs.
Design realistic evaluation environments and task sets reflecting business work performed by AI agents.
Develop accurate, reproducible, and difficult-to-game scoring methods using automated checks, LLM-based judges, and human review; measure consistency and reliability.
Audit evaluation tasks and graders, quantify noise, and establish whether differences between agents are meaningful.
Present and defend evaluation findings to buyers, customers, partners, and the research community.
Mentor and guide researchers and engineers, review evaluation design and methodology, and create documentation and standards that enable the team to run evaluations consistently.
Set and pursue a research agenda, selecting important problems and carrying long-running projects through to practical outcomes.
Follow research papers, benchmarks, and evaluation practices, translating relevant developments into product and roadmap decisions and communicating them to the team.
Write methodology documentation and, where appropriate, papers and public posts; collaborate with data and go-to-market teams on presenting and using results.