AI Evaluation Specialist
About The Position
Chenega Services & Federal Solutions, LLC, a Chenega Professional Services company, is seeking an AI Evaluation Specialist to assess enterprise AI tools and capabilities for effectiveness, safety, usability, and readiness for business adoption. The specialist will work with product managers, data scientists, AI and software engineers, security and privacy teams, and business stakeholders to define evaluation criteria and conduct structured assessments of generative AI products and platforms, including OpenAI tools and life sciences models such as Rosalind. The role involves designing repeatable evaluation frameworks, testing real-world use cases, analyzing model and product behavior, and turning findings into clear recommendations. Evaluations compare AI solutions across quality, accuracy, reliability, safety, privacy, and security. This position is contingent on contract award, and the details may change based on the final award. The position is remote within the continental United States. Final salary depends on skill set, qualifications, and approved funding.
Requirements
- At least 5 years of experience in model evaluation or model quality assurance, including hands-on evaluation of AI-enabled products or services.
- Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria.
- Hands-on experience with generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identification of hallucinations and other failure modes.
- Experience assessing AI systems for accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost.
- Familiarity with generative AI architectures and capabilities, including large language models, RAG, AI agents, APIs, enterprise search, and tool or connector integrations.
- Bachelor’s degree.
Nice To Haves
- Experience evaluating enterprise AI platforms or assistants, including OpenAI products and life sciences models such as Rosalind, or comparable generative AI solutions.
- Experience with automated LLM evaluation frameworks, model-as-a-judge approaches, human-in-the-loop evaluation, red teaming, or adversarial testing.
- Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows.
- Experience developing dashboards or reporting mechanisms for AI evaluation metrics, quality trends, risk findings, and regression results.
- Ability to work with datasets and evaluation results using Python, SQL, spreadsheets, notebooks, or analytics platforms to calculate metrics, identify patterns, and summarize findings.
- Industry-recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI.
- Master’s degree.
Responsibilities
- Plan and conduct hands-on evaluations of AI tools using representative business scenarios, workflows, prompts, datasets, and user personas.
- Assess response quality, factual accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and overall task effectiveness.
- Design and maintain standardized evaluation frameworks, scorecards, rubrics, and acceptance criteria for enterprise AI tools, models, agents, and AI-enabled applications.
- Evaluate safety, privacy, security, data handling, governance, and responsible AI considerations with relevant stakeholders.
- Compare AI products, models, configurations, and vendors, including functionality, integration requirements, performance, and cost.
- Evaluate AI agents and tool-enabled workflows, including RAG, enterprise search, file and data analysis, connectors, APIs, and MCP-enabled capabilities where applicable.
- Keep current with developments in generative AI, model evaluation, AI safety, responsible AI, and enterprise AI products, and incorporate relevant methods into the evaluation program.
- Share knowledge and contribute to a collaborative team environment.
Benefits
- Paid time off and vacation accrual (5.67 hours per pay period; 136 hours annually).
- 11 paid holidays.
- California residents receive an additional 24 hours of sick leave per year.
- Medical insurance.
- Dental insurance.
- Vision insurance.
- Prescription coverage.
- Employee Assistance Program.
- Short- and long-term disability insurance.
- Life and AD&D insurance.
- Flexible Spending Account.
- Health Savings Account.
- Health Reimbursement Account.
- Dependent Care Spending Account.
- Commuter benefits.
- 401(k) / 401(a) retirement plans.
- Hospital indemnity coverage.
- Critical illness coverage.
- Accident insurance.
- Pet insurance.
- Legal insurance.
- ID theft protection.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
