AI Evaluation Engineer

Judi Health•Charlotte, NC

19h

About The Position

As an AI Evaluation Engineer at Judi Health, you will build the testing frameworks, metrics, and tooling used to assess the safety, reliability, and accuracy of AI models and autonomous agents in production. This role bridges the gap between model development and real‑world usage by translating ambiguous product goals into measurable quality targets. We’re looking for someone to lead evaluation end-to-end — from unit and integration testing to offline, online, and statistical evaluations of probabilistic systems. What we need is someone who can design and operate robust evaluation frameworks, partner with scientists and engineers, and ensure we can confidently answer questions like: “Did this change improve or degrade quality, safety, or user outcomes?”

Requirements

4+ years of experience in data engineering, ML engineering, or software engineering
Bachelor’s or Master’s degree in Computer Science, Machine Learning, or a related quantitative field
Strong proficiency in Python
Experience building and maintaining production data pipelines
Strong SQL skills
Experience working with at least one cloud platform (AWS preferred)

Nice To Haves

Prior work on LLM or agent evaluation infrastructure
Familiarity with designing metrics for safety, reliability, or quality in AI systems
Experience with voice or call‑center data (audio, transcripts, sentiment)
Experience with browser automation tools (e.g., Playwright) for end‑to‑end evals
Deep SQL expertise

Responsibilities

Build data evaluation pipelines that collect production conversations and agent interactions
Reconstruct full sessions from traces, logs, recordings, and transcripts
Apply labeling and scoring using human feedback signals (surveys, sentiment, outcomes) and automated evaluators (e.g., LLM‑as‑judge)
Own weekly and on‑demand automated evaluation runs against staging and production
Define benchmarks that track accuracy, reliability, and safety‑related signals
Produce trend dashboards that clearly answer: “Did this deploy change quality or risk?”
Design and extend a standardized evaluation framework that supports multiple agent types and workflows
Translate high‑level product expectations into concrete success criteria and metrics
Ensure new agents and features can be evaluated consistently with minimal friction
Build APIs and internal tools so data scientists and engineers can go from “interesting scenario” to “included in the eval suite” quickly
Enable scenario curation, dataset management, and eval execution without deep infrastructure knowledge
Provide shared visibility into prompt, model, and agent experiments
Enable reproducibility and comparison across runs so teams can build on each other’s work instead of operating in silos
Build and maintain ETL pipelines for heterogeneous data sources (traces, logs, transcripts, user feedback)
Implement complex data stitching and session reconstruction logic
Manage dataset versioning, provenance, and lifecycle
Develop dashboards and monitoring tools for AI quality metrics
Integrate evaluations into CI/CD pipelines for scheduled and gated runs
Implement alerting on quality and safety signals, not just infrastructure health
Apply and extend LLM‑as‑judge evaluation patterns
Design metrics and scoring approaches suitable for stochastic, non‑deterministic systems
Use tools like LangSmith to track runs, traces, experiments, and evaluation results
Partner closely with data science, engineering, and product teams
Translate between research goals, product intent, and engineering constraints
Help define what “good” looks like for AI behavior in production
Advocate for strong developer experience and usability in the tools you build
Responsible for adherence to the Capital Rx Code of Conduct including the reporting of non-compliance.