AI Evaluation Scientist

Steampunk•McLean, VA
•$105,000 - $145,000

About The Position

The AI Evaluation Scientist designs and executes evaluation processes to ensure predictive and generative AI systems are accurate, reliable, safe, and aligned with mission requirements. The role establishes trust in AI solutions and supports continuous improvement across the AI lifecycle. It works closely with engineers, data scientists, governance analysts, and product teams to develop evaluation metrics, build test harnesses, analyze model behavior, and support responsible deployment. The position contributes to the growth of Steampunk’s AI & Data Exploitation Practice. It involves evaluating AI systems, documenting findings for technical and non-technical audiences, and supporting evaluation outputs used in compliance, documentation, and risk assessment.

Requirements

  • Ability to hold a position of public trust with the U.S. government.
  • Bachelor’s degree and 8+ years of work experience, or an equivalent combination of education and experience.
  • At least 2 years of experience evaluating machine learning models, NLP systems, or generative AI models; LLM experience is preferred.
  • Familiarity with evaluation metrics, statistical testing, dataset creation, and experimental design for AI systems.
  • Proficiency in Python and relevant libraries such as PyTorch, Hugging Face, scikit-learn, and LangChain.
  • Proficiency in AI evaluation frameworks such as Ragas or DeepEval.
  • Proficiency in AI traceability/observability tools based on the OpenTelemetry protocol.
  • Experience analyzing structured and unstructured data, including text, documents, and embeddings.
  • Understanding of LLM behavior, prompt evaluation, retrieval pipelines, or RAG architectures.
  • Exposure to responsible AI concepts and governance-aligned evaluation criteria, such as fairness, transparency, and reliability.
  • Strong analytical skills, including the ability to interpret model weaknesses, extract insights, and recommend actionable improvements.
  • Excellent written and verbal communication skills, including the ability to present evaluation findings clearly to technical and non-technical stakeholders.

Nice To Haves

  • Experience working in agile or iterative development environments.
  • Familiarity with OWASP LLM Top 10 Risks.
  • Relevant certifications are helpful but not required: NIST AI RMF (AISIC), INFORMS CAP, or AWS, Azure, or Google ML certifications.

Responsibilities

  • Implement evaluation frameworks for AI models, including accuracy, robustness, relevance, bias, hallucination rate, and safety metrics.
  • Build and maintain automated evaluation scripts, tests, and pipelines to assess AI model outputs and detect performance drift over time.
  • Develop benchmark datasets, challenge sets, and scenario-based test cases tailored to mission and user needs.
  • Perform structured error analysis and behavioral audits of LLMs, retrieval-augmented generation (RAG) systems, and predictive models, documenting findings and improvement recommendations.
  • Collaborate with AI Developers, LLMOps Engineers, and Data Scientists on iterative experimentation, model hardening, and quality improvements.
  • Contribute to the design of human-in-the-loop evaluation workflows, integrating qualitative and quantitative insights into evaluation reports.
  • Help map evaluation outcomes to responsible AI principles such as fairness, transparency, reliability, and safety.
  • Partner with AI Governance Analysts to ensure evaluation outputs support compliance, documentation, and risk assessments.
  • Stay current with emerging evaluation tools, frameworks, metrics, and research related to LLM assessment and generative AI reliability.
  • Document evaluation processes, criteria, and results for technical and non-technical audiences.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service