Model Test and Measurement Engineer

DEFCON AI

18h • $150,000 - $190,000 • Remote

About The Position

DEFCON AI is an insights company that leverages artificial intelligence, mathematical optimization, data analytics, and software engineering for resilient optimization of complex systems. In today’s dynamically changing world, DEFCON AI’s technology aligns outcomes with operational goals, better decision making, and empowers customers to anticipate assess, and mitigate the impacts of disruptions. The analytics and AI engineering team is building an AI-assisted platform that brings together records from dozens of disparate data sources, resolves them to the correct individual, highlights what analysts should review first, and provides transparent, explainable recommendations that users can trust. Operating within a secure government cloud environment, the platform tackles complex challenges in AI, data integration, and decision support where quality, trust, and accountability are mission-critical. As a Model Test & Measurement Engineer, you'll own the evaluation framework that helps ensure those systems perform as intended. You'll build and maintain labeled ground truth datasets, design statistically sound audit and sampling methodologies, measure model performance across releases, and create the evidence packages that support deployment decisions. You'll independently validate both scoring and generative AI capabilities, helping the team understand not only whether a model works, but how confidently its outputs can be trusted. Your assessments will inform release decisions, drive improvement efforts, and provide the objective evidence customers rely on when evaluating system performance. You'll work closely with data scientists, AI engineers, and technical leadership while maintaining the independence needed to provide clear, defensible evaluations. This is a fully remote role with occasional travel to DEFCON AI headquarters, customer sites, and partner facilities as needed.

Requirements

• 5+ years with model validation as a named responsibility, not a side task
• Direct experience with ground-truth construction and sampling design
• Strong Python and SQL; comfort working independently from the teams whose models you evaluate
• US Citizenship Required
• Active US Secret clearance
• Elevated personnel security requirements apply to portions of this work and are discussed during screening

Nice To Haves

• Regulated-industry or government model-risk background
• Experience with fairness / disparate-effect testing and human-oversight documentation
• NIST AI RMF or comparable practice
• Active Top Secret clearance

Responsibilities

• Construct labeled ground truth for model evaluation
• Design and run sampled audits
• Gate releases against model version; maintain version inventory, evaluation records, and rollback triggers
• Run drift and override review
• Produce human-oversight and fairness / disparate-effect evidence
• Maintain independence from the build roles: produce evaluation evidence, but do not build the models, approve thresholds, or authorize releases against that evidence
• Measure workflow improvement: review time, throughput, backlog movement, and override and rework rates
• Design evaluation-phase QC: sample selection that does not mix populations, the unit of review, and fair comparison when methods or searched sources differ
• Define the measurement events other teams must emit, and confirm the review workspace emits workflow events from first use

Benefits

• Competitive salary, bonus, and equity package
• 100% employer paid, comprehensive health insurance including medical, dental, and vision for you and your family
• Unlimited PTO, with your manager's approval
• Flexible work environment where you manage your work day
• 14 weeks of fully-paid parental leave

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed