Data Scientist

Precision Solutions • Fort Meade, MD

3d • Onsite

About The Position

Since 2012, our client has helped mission-critical government organizations and businesses face their most daunting technology challenges. Their team have been trusted partners to many government agencies and are extremely familiar with a wide variety of systems, policies, and procedures. Our client is also a distinguished custom software development firm dedicated to delivering premium solutions tailored for businesses and governmental needs. They are home to top-tier technology professionals recognized as industry pioneers, comprehensive full-stack engineers, and reliable consultants. These experts are adept at clear communication, excel in resolving complex challenges where others may falter, and are skilled in actualizing an organization's vision.
The Data Scientists' primary responsibility is to design, build, and maintain benchmarks for model testing and agentic workflow testing using InspectAI. They define evaluation datasets, test scenarios, metrics, scoring approaches, and repeatable evaluation workflows that measure model and agent performance in a consistent, auditable way. They also clean, transform, validate, and analyze supporting data; generate metrics and visualizations; and produce findings that support model evaluation, reporting, and decision-making. They partner with Researchers to ground benchmark design in sound methodology, and with Developers to operationalize evaluation workflows into stable, repeatable tooling. Data Scientists should also be able to document dataset lineage, scoring rationale, benchmark assumptions, and evaluation outputs so results are reproducible, traceable, and usable in constrained or airgapped environments.

Requirements

• Experience with AI Evaluation & Frameworks such as InspectAI, LLM Benchmarking, Agentic Workflow Evaluation (tool-use, multi-step task performance)
• Experience with Statistical Analysis, Data Cleaning & Validation, Data Visualization Tools, Jupyter Notebooks
• Experience with MongoDB, Cloud/Containerized Environments, Airgapped & Constrained Operations
• Familiarity with Git Version Control, Experimental Design, Dataset Lineage/Traceability, Jira, and Confluence
• BS in Computer Science or similar technical field (4 years of relevant experience may be substituted in lieu of a BS degree)
• Active TS/SCI w/FS Poly (NSA) Clearance Required

Responsibilities

• Design, implement, and maintain benchmark suites that measure model performance against defined evaluation criteria, including dataset curation, scoring, and results reporting.
• Develop benchmarks and test scenarios that evaluate agentic workflows and multi-step task performance, including tool-use, task completion, and reliability metrics.
• Curate, validate, and version evaluation datasets and test scenarios, including any adversarial or edge-case sets needed to stress agent behavior.
• Produce scoring frameworks, visualizations, and written analyses that translate raw evaluation results into actionable findings for stakeholders.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior