AI Evaluation Specialist

Chenega Corporation•UNAVAILABLE, Georgia

1d•Remote

About The Position

The AI Evaluation Specialist evaluates enterprise AI tools and capabilities to determine their effectiveness, safety, usability, and readiness for business adoption. The role partners with product managers, data scientists, AI engineers, software engineers, security and privacy teams, and business stakeholders to define evaluation criteria and conduct structured assessments of generative AI products and platforms, including tools such as OpenAI and life sciences models like Rosalind. The specialist designs repeatable evaluation frameworks, tests real-world use cases, analyzes model and product behavior, and translates findings into clear recommendations. The role compares AI solutions across quality, accuracy, reliability, safety, privacy, and security. This position is contingent on contract award, and details may change based on the final award.

Requirements

At least 5 years of experience in model evaluation or model quality assurance, with hands-on experience evaluating AI-enabled products or services.
Experience developing structured evaluation plans, test cases, scoring rubrics, benchmarks, and measurable acceptance criteria.
Hands-on experience with generative AI tools and large language models, including prompt design, systematic testing, output analysis, and identifying hallucinations and other failure modes.
Experience assessing AI systems for accuracy, relevance, consistency, safety, bias, privacy, security, usability, performance, and cost.
Ability to demonstrate familiarity with common generative AI architectures and capabilities, including large language models, RAG, AI agents, APIs, enterprise search, and tool or connector integrations.
Bachelor's degree.

Nice To Haves

Experience evaluating enterprise AI platforms or assistants, including OpenAI products and life sciences models like Rosalind or comparable generative AI solutions.
Experience with automated LLM evaluation frameworks, model-as-a-judge approaches, human-in-the-loop evaluation, red teaming, or adversarial testing.
Experience evaluating RAG systems, AI agents, MCP-enabled tools, enterprise connectors, or multi-step AI workflows.
Experience developing dashboards or reporting mechanisms for AI evaluation metrics, quality trends, risk findings, and regression results.
Ability to work with datasets and evaluation results using tools such as Python, SQL, spreadsheets, notebooks, or analytics platforms to calculate metrics, identify patterns, and summarize findings.
Industry-recognized certification in AI, machine learning, data science, cybersecurity, cloud technologies, or responsible AI.
Master’s degree.

Responsibilities

Plan and execute hands-on evaluations of AI tools, including OpenAI and life sciences models like Rosalind, using representative business scenarios, workflows, prompts, datasets, and user personas.
Assess AI solutions for response quality, factual accuracy, relevance, consistency, reasoning, instruction following, reliability, latency, usability, accessibility, and overall task effectiveness.
Design and maintain standardized evaluation frameworks, scorecards, rubrics, and acceptance criteria for enterprise AI tools, models, agents, and AI-enabled applications.
Evaluate safety, privacy, security, data handling, governance, and responsible AI considerations with security, privacy, legal, and governance stakeholders.
Conduct comparative evaluations and proof-of-concept assessments across AI products, models, configurations, and vendors, including functionality, integration requirements, performance, and cost.
Evaluate AI agents and tool-enabled workflows, including retrieval-augmented generation (RAG), enterprise search, file and data analysis, connectors, APIs, and Model Context Protocol (MCP)-enabled capabilities where applicable.
Stay current with advancements in generative AI, model evaluation, AI safety, responsible AI, and emerging enterprise AI products, and incorporate relevant methods into the evaluation program.
Engage in knowledge sharing and contribute to a strong, collaborative team environment.