Senior AI/ML Test and Evaluation Engineer

OpenTeams•Denver, WA

7d•Hybrid

About The Position

We're looking for a Senior AI/ML Test and Evaluation Engineer to build and operate the benchmarking and evaluation capability at the core of an AI platform. This is a role for someone who is more interested in what a model gets wrong than in what it gets right. You build the evaluation harnesses — automated metrics paired with structured human expert judgment, applied to candidate models and to the agentic workflows built on top of them. You develop repeatable methodologies for comparing performance against current operational baselines, which means the comparison holds up when someone runs it again in six months with a different model. And you document the limitations and surface the failure modes that matter, including the ones nobody asked about. Your reports go to senior stakeholders and inform decisions about which capabilities are ready to field. That's the weight of the job: a benchmark that looks good and hides a failure mode is worse than no benchmark at all, and you're the check against that. This is hands-on engineering on an open-source toolchain. This position is contingent upon contract award. Travel of up to 15% may be required, primarily to Government facilities and between company locations.

Requirements

U.S. citizenship and eligibility to obtain and maintain a U.S. security clearance
6+ years of software engineering or machine learning engineering experience, including 3+ years evaluating, benchmarking, or deploying ML models in production or applied research environments
Strong Python proficiency in a machine learning or data science context
Hands-on experience with common ML frameworks and tooling, such as PyTorch and the Hugging Face ecosystem
Experience developing or using model evaluation harnesses, benchmark suites, or test and evaluation frameworks
Experience designing evaluation metrics and applying appropriate statistical rigor when interpreting and reporting results
Experience building repeatable and auditable evaluation pipelines with documented data provenance
Experience evaluating large language models or agentic workflows using task-based, metric-based, or judgment-based scoring
Strong written communication skills, including the ability to clearly explain evaluation methodologies, results, limitations, and failure modes to technical and nontechnical stakeholders
Ability to work effectively in an evolving environment and translate mission needs into practical evaluation approaches
Bachelor’s degree in computer science, mathematics, engineering, or a related field, or equivalent practical experience

Nice To Haves

Active U.S. security clearance
Prior AI/ML evaluation or test and evaluation experience supporting the Department of Defense, Intelligence Community, or another federal customer
Experience designing human-in-the-loop evaluations, measuring inter-rater reliability, or facilitating structured expert adjudication
Experience defining or implementing benchmark interchange formats or evaluation standards used across multiple organizations
Familiarity with intelligence analysis workflows or other high-stakes analytical domains
Experience working directly with Government stakeholders, mission users, or external technical partners
Contributions to open-source machine learning, benchmarking, or evaluation projects

Responsibilities

Design, implement, and operate benchmark execution and evaluation harnesses for AI models and agentic workflows
Develop evaluation methodologies that combine automated metrics with structured human subject matter expert judgment
Curate and recommend candidate benchmarks based on mission needs and document the provenance of ground-truth and reference data
Produce defensible evaluation reports comparing candidate capabilities with current mission workflows, including documented limitations and failure modes
Define and contribute to common standards for benchmark expression, ingestion, and reporting
Support partner organizations and vendors as they integrate their capabilities with shared evaluation standards
Build lightweight expert-scoring workflows and measure inter-reviewer agreement for judgment-based evaluations
Participate in structured feedback sessions with mission end users and incorporate findings into the platform and evaluation methodology
Develop reference notebooks and example workflows that enable data-science-capable analysts to run, interpret, and extend evaluations
Document technical approaches, evaluation results, and key decisions for Government stakeholders and internal teams