About The Position

Solvd Inc. is seeking a Senior AI Quality & Evaluation Engineer to independently verify retrieval quality and access permissions for AI systems. The role will build and maintain acceptance-gating evaluations, including golden-question suites, negative entitlement tests, freshness and retirement checks, and drift alarms. It is a 50% allocation role across weeks 3–10, drawn from a Quality Engineering or GRC background. The position reports to the POD Lead and works day-to-day with the retrieval engineer. The engineer will automate scheduled and change-triggered evaluation runs, publish scores and regression trends to a quality dashboard, and report test failures with root causes to the appropriate owners. The role also defines QA gates for remediated content and creates an evaluation runbook, while training client stewards and platform owners to operate and interpret the evaluation suite.

Requirements

  • At least 3 years of experience in software quality engineering or test automation with Python, including building test harnesses that run in CI on a schedule.
  • Hands-on experience evaluating a search or RAG system in production, including creating ground-truth sets, scoring relevance and grounding, and tracking drift over time.
  • Experience designing security-focused negative tests for authorization, entitlement, and data exposure, and reporting findings at the severity expected by a client security team.
  • Ability to work in AWS, including invoking Bedrock and OpenSearch APIs, reading CloudWatch, and deploying a scheduled job with infrastructure code.
  • Clear written reporting skills, as client stewards as well as engineers read the evaluation suite's output.

Nice To Haves

  • AWS Certified AI Practitioner or Machine Learning Engineer Associate certification.
  • Experience with RAG evaluation libraries and LLM-as-judge scoring with human calibration.
  • Contact center or knowledge management domain exposure.
  • Familiarity with SOC 2 or ISO 27001 evidence practices.

Responsibilities

  • Build a versioned golden-question suite from top contact drivers and needs workshops, specifying each question, expected source and passage, and acceptance thresholds for relevance, grounding, and citation accuracy.
  • Build negative entitlement tests that verify queries under unauthorized identities return no restricted passages, and raise any failure as a severity-one alarm.
  • Build freshness and retirement tests measuring source-modified-to-index-available lag against agreed thresholds and verifying retired items disappear on the next sync.
  • Automate the evaluation harness to run on a schedule and after content, configuration, or model changes, publishing scores, trends, and regression alarms to the quality dashboard.
  • Define the QA gate content analysts use to assess remediated content.
  • Run the evaluation suite before every pilot change in weeks 7–9 and report misses and root causes—content, chunking, ranking, or permissions—to the appropriate owner.
  • Write the evaluation runbook and train client stewards and platform owners to add questions, rerun the suite, and interpret regressions.

Benefits

  • Equal opportunities for collaboration across continents and cultures.
  • Inclusive environment that prioritizes continuous learning, innovation, and ethical AI standards.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service