Evaluation Engineer

Freshfields Bruckhaus Deringer • New York, NY

1d • $100,000 - $200,000

About The Position

The Evaluation Engineer builds and runs tests to assess the quality and reliability of the firm’s AI applications. The role helps maintain evaluation datasets, investigate failures, and provide clear evidence of AI system performance on legal and business tasks.

The engineer implements automated evaluation cases and supporting scripts using established frameworks and guidance from senior engineers. The role also prepares and maintains test datasets, source documents, and reference examples in collaboration with legal specialists, following confidentiality, permitted-data-use, and version-control requirements.

The position evaluates answer accuracy, completeness, citations, retrieval relevance, instruction following, and appropriate escalation using agreed scoring criteria. It includes functional and integration testing of AI workflows; reviewing outputs and execution traces to identify, reproduce, and document failures; comparing results across model, prompt, and application versions; and reporting changes with appropriate attention to uncertainty and limitations.

The Evaluation Engineer supports calibration of automated graders against human reviews, maintains regression suites, contributes to CI/CD integration, and converts approved production failure examples into new test cases. The role requires at least five years of software development experience and a foundational understanding of LLMs, statistics, and measurement. Practical AI application evaluation experience is a plus, but is not required.

Requirements

• At least five years of software development experience demonstrated through employment, placements, or substantial projects using languages such as Python, Java, or C#.
• Foundational conceptual understanding of large language models, including capabilities, limitations, and common failure modes, and ability to apply that understanding when developing evaluation criteria, test cases, and acceptance thresholds with guidance from senior engineers.
• Ability to use coding agents effectively for development, testing, and analysis, with awareness of prompting, context management, validation, security, responsible use of generated code, and when to seek human review.
• Working knowledge of APIs, Git, automated testing, software development lifecycle practices, and CI/CD tools such as Jira and Jenkins.
• Curiosity and ability to learn in a rapidly evolving AI landscape, assess new models, tools, and evaluation techniques, and apply relevant learning pragmatically with appropriate guidance.
• Foundational understanding of statistics and measurement, including sampling, precision, and recall.
• Strong analytical skills and attention to detail, with the ability to document issues clearly and seek technical or legal guidance when needed.

Nice To Haves

• Practical experience evaluating AI applications; relevant academic, internship, or project experience will also be considered.

Responsibilities

• Implement automated evaluation cases and supporting scripts using established frameworks and guidance from senior engineers.
• Prepare and maintain test datasets, source documents, and reference examples with legal specialists while following confidentiality, permitted data use, and version-control requirements.
• Evaluate answer accuracy, completeness, citations, retrieval relevance, instruction following, and appropriate escalation using agreed scoring criteria; flag cases requiring expert judgment.
• Execute functional and integration tests for AI workflows, including document ingestion, tool calls, structured outputs, and error handling, in coordination with application QA.
• Review outputs and execution traces to identify failures, reproduce issues, document supporting evidence, and help distinguish data, retrieval, model, and application defects.
• Compare results across model, prompt, and application versions, reporting changes accurately and describing uncertainty and limitations.
• Support calibration of automated graders by comparing their scores with human reviews and investigating inconsistencies.
• Maintain regression suites, contribute to CI/CD integration, and turn approved production failure examples into new test cases.
• Assess new models, tools, and evaluation techniques and apply relevant learning pragmatically with appropriate guidance.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: No Education Listed