QA / Evaluation Lead
About The Position
Innodata is a global data engineering company focused on enabling the responsible advancement of artificial intelligence by providing the data, evaluation frameworks, and human expertise needed to build trustworthy AI systems. The Federal Practice specifically builds the trusted data layer for critical infrastructure Trust & Safety work within a secure federal (IL4) environment. This role will be instrumental in standing up a data services storefront, a DataCard governance framework, synthetic data integration, and Databricks write-back capabilities over an intensive 20-week phase. As the QA/Evaluation Lead, you will be responsible for quality and evaluation across the platform. This includes designing the evaluation framework to measure the quality of data services and outputs, building repeatable test and validation processes, and providing objective assessments of readiness at each milestone. You will collaborate with the Delivery Owner and engineering leads to establish quality as a measurable and demonstrable strength, thinking rigorously about evaluation and valuing evidence-backed quality.
Requirements
- Bachelor's degree in Statistics, Data Science, Computer Science, or related quantitative field required; Master's degree preferred. Equivalent experience may substitute for degree on a 2-for-1 basis.
- 6+ years total professional experience, 4+ years in data quality, evaluation methodology, or QA on AI/ML programs
- IAA methodology expertise — Cohen's kappa, Fleiss' kappa, Krippendorff's alpha: hands-on, not theoretical
- Evaluation framework design for AI/ML training data programs
- QC process design: sampling methodology, escalation workflows, adjudication protocols
- Python for QC tooling, metric computation, and statistical analysis
- Active Secret clearance with TS/SCI eligibility
Nice To Haves
- Prior DoD or IC data quality program experience
- CVAT or equivalent annotation platform QC workflow configuration
- Drift detection and model monitoring methodology
- Experience with FMV / video annotation quality standards
Responsibilities
- Design and own the inter-annotator agreement (IAA) methodology for the Phase 1 demonstration corpus — metric selection (Cohen's kappa, Fleiss, Krippendorff's alpha), sampling design, adjudication workflow, and agreement thresholds
- Define evaluation framework architecture: test and evaluation plans, IAA targets, drift detection gates, and model performance metrics per SOW Section 2.9
- Configure and operate sampling-based quality control across the self-service and white-glove annotation paths during Phase D corpus production
- Design and implement confidence-threshold escalation routing from automated annotation to senior-annotator adjudication
- Validate quality scoring and IAA computation within the Innodata data layer
- Support AI Solutions Engineer on evaluation design for SAM 2 and Frontier model API validation — define what 'good enough' looks like quantitatively
- Produce evaluation framework documentation for the Phase 1 NPP closeout package, including per-DataCard documentation with the SA
Benefits
- The expected hourly salary range for this position is $45 to $50 p/hour, based on experience, skills, and qualifications.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
