AI Evaluation Scientist

Steampunk • McLean, VA

2d • $105,000 - $145,000

About The Position

In today’s rapidly evolving technology landscape, an organization’s data has never been a more important aspect in achieving mission and business goals. Our AI & Data Exploitation experts work with our clients to support their mission and business goals by creating and executing a comprehensive data strategy using the best technology and techniques, given the challenge. At Steampunk, our goal is to build and execute a data strategy for our clients to coordinate data collection and generation, to align the organization and its data assets in support of the mission, and ultimately to realize mission goals with the strongest effectiveness possible. For our clients, data is a strategic asset. They are looking to become a facts-based, data-driven, customer-focused organization. To help realize this goal, they are leveraging visual analytics platforms to analyze, visualize, and share information. At Steampunk you will design and develop solutions to high-impact, complex data problems, working with the best and data practitioners around. Our data exploitation approach is tightly integrated with Human-Centered Design and DevSecOps.

Requirements

• Ability to hold a position of public trust with the U.S. government.
• Bachelor’s or Master’s degree in Computer Science, Statistics, Machine Learning, Cognitive Science, Human-Computer Interaction, Data Science, or a related field.
• 2+ years of experience evaluating machine learning models, NLP systems, or generative AI models (LLMs preferred).
• Familiarity with evaluation metrics, statistical testing, dataset creation, and experimental design for AI systems.
• Proficiency in Python and relevant libraries such as PyTorch, Hugging Face, scikit-learn, LangChain.
• Proficiency in AI evaluation frameworks such as Ragas or DeepEval.
• Proficiency in AI traceability/observability tools based on the OpenTelemetry protocol.
• Experience analyzing structured and unstructured data, including text, documents, and embeddings.
• Understanding of LLM behavior, prompt evaluation, retrieval pipelines, or RAG architectures.
• Exposure to responsible AI concepts and governance-aligned evaluation criteria (e.g., fairness, transparency, reliability).
• Strong analytical skills with the ability to interpret model weaknesses, extract insights, and recommend actionable improvements.
• Excellent written and verbal communication skills, with the ability to present evaluation findings clearly to technical and non-technical stakeholders.

Nice To Haves

• Experience working in agile or iterative development environments is a plus.
• Familiarity with OWASP LLM Top 10 Risks
• Relevant certifications (helpful but not required): NIST AI RMF (AISIC), INFORMS CAP, AWS/Azure/Google ML Certifications.

Responsibilities

• Implement evaluation frameworks for AI models, including accuracy, robustness, relevance, bias, hallucination rate, and safety metrics.
• Build and maintain automated evaluation scripts, tests, and pipelines that assess AI model outputs and detect performance drift over time.
• Develop benchmark datasets, challenge sets, and scenario-based test cases tailored to mission and user needs.
• Perform structured error analysis and behavioral audits of LLMs, retrieval-augmented generation (RAG) systems, and predictive models, documenting findings and improvement recommendations.
• Collaborate with AI Developers, LLMOps Engineers, and Data Scientists to support iterative experimentation, model hardening, and quality improvements.
• Contribute to the design of human-in-the-loop evaluation workflows, integrating qualitative and quantitative insight into evaluation reports.
• Assist in mapping evaluation outcomes to responsible AI principles such as fairness, transparency, reliability, and safety.
• Partner with AI Governance Analysts to ensure evaluation outputs support compliance, documentation, and risk assessments.
• Stay current with emerging evaluation tools, frameworks, metrics, and research related to LLM assessment and generative AI reliability.
• Document evaluation processes, criteria, and results for both technical and non-technical audiences.
• Contribute to the growth of our AI & Data Exploitation Practice!

Benefits

• As an employee owned company, we focus on investing in our employees to enable them to do the greatest work of their careers – and rewarding them for outstanding contributions to our growth.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level