AI Evaluation Engineer

Sedgwick•Telecommuter FL, FL

2d•Remote

About The Position

The AI Evaluation Engineer designs, builds, and operates evaluation systems to assess whether GenAI solutions are accurate, safe, reliable, and ready for production. Serving multiple product pods, the role establishes consistent evaluation standards, automated pipelines, and quality gates, and helps teams measure and improve LLM, RAG, and agentic applications throughout their lifecycle. This is a fully remote role open to U.S.-based candidates.

Requirements

At least 4 years of experience in software, ML, or data engineering, including at least 1–2 years of hands-on work with LLM-based applications.
Professional experience with evaluation and observability frameworks such as LangSmith, Arize Phoenix, or WhyLabs.
Strong understanding of statistical validation and model evaluation metrics, and ability to partner with Data Scientists to optimize model performance.
Strong Python skills and software engineering fundamentals, including testing, version control, CI/CD, and code review.
Practical experience evaluating LLM/RAG systems, including retrieval metrics such as recall, precision, MRR/nDCG, and generation metrics.
Hands-on experience designing LLM-as-a-judge systems, prompt engineering, and judge validation.
Working knowledge of statistics and experimental design, including sampling, confidence intervals, significance testing, and inter-rater agreement.
Familiarity with at least one evaluation or observability tool, such as Ragas, DeepEval, promptfoo, LangSmith, Langfuse, MLflow, or an equivalent.
Experience with cloud platforms such as Azure, AWS, or GCP and their managed LLM services.
Clear communication skills and ability to work with engineers, product owners, subject-matter experts, and risk stakeholders.
Excellent oral and written communication skills, including presentation skills.
Ability to work in a team environment and meet or exceed Performance Competencies.
Ability to meet deadlines, manage multiple priorities, exercise judgment and discretion, and perform troubleshooting, problem-solving, and analysis.

Nice To Haves

Master’s degree in computer science or a quantitative discipline from an accredited college or university.
Experience evaluating agentic systems, including multi-step reasoning, tool calling, and trajectory evaluation.
Background in ML evaluation, search/relevance, NLP, or QA automation.
Experience with LLMOps, monitoring and observability, and production feedback loops.
Familiarity with AI risk and governance frameworks such as NIST AI RMF or ISO/IEC 42001, and secure development practices in regulated environments.
Experience with data annotation workflows and working with subject-matter experts.
Knowledge of enterprise controls such as SSO, RBAC, private networking, logging, and encryption.

Responsibilities

Establish standardized, automated evaluation frameworks for LLMs and traditional ML models, using approaches such as LLM-as-a-judge, ROUGE, METEOR, and statistical validation.
Deploy observability solutions to monitor model drift, hallucination risk, latency, throughput, and token consumption while balancing quality, infrastructure cost, and operational efficiency.
Define evaluation methodologies and metrics for LLM, RAG, and agent-based applications, including faithfulness, groundedness, relevance, task completion, tool-use correctness, and safety.
Build reusable evaluation frameworks, templates, and libraries that product pods can adopt.
Partner with product owners and subject-matter experts to translate business requirements into measurable quality criteria and acceptance thresholds.
Develop automated evaluation pipelines integrated into CI/CD, including regression testing and quality gates for prompt, model, and retrieval changes.
Implement offline evaluation using benchmark and golden datasets, and online evaluation using production monitoring, sampling, drift detection, and quality alerts.
Support A/B and shadow testing for model, prompt, and architecture changes.
Design, validate, and maintain LLM-as-a-judge evaluators, including rubrics, bias mitigation, and calibration against human labels.
Establish human annotation workflows, labeling guidelines, and inter-annotator agreement practices with subject-matter experts.
Track judge reliability and recalibrate evaluators as models and use cases change.
Build and curate golden datasets, adversarial and edge-case suites, and approaches for generating synthetic test data.
Apply approved data-handling practices, including de-identification, synthetic data, and access controls, to datasets derived from sensitive or regulated data.
Implement red-teaming and testing for hallucinations, toxicity, PII leakage, prompt injection, and jailbreaks.
Produce evaluation evidence and reporting to support security, privacy, compliance, and model risk reviews.
Contribute to responsible AI standards and release-readiness criteria.
Embed with product pods to introduce evaluation practices and coach teams toward ownership of their evaluation suites.
Build dashboards and reporting that show engineering and business leaders quality, cost, and latency trends.
Evaluate and recommend evaluation tooling and platforms, and document standards and best practices.
Perform other duties as assigned and travel as required.