AI Quality Analyst, AI & Innovation

Lockton•Kansas City, MO

20h

About The Position

The AI Quality Analyst, AI & Innovation safeguards the quality, reliability, and measurable performance of AI data extraction processes that convert commercial risk insurance documents into structured data. The role supports the organization’s Enterprise AI Evaluation Strategies by building and maintaining ground-truth datasets and contributing to the evaluation processes those datasets support. It does not require hands-on AI/ML development experience. Reporting to the Director, AI & Innovation, the analyst creates, curates, and maintains ground-truth files; helps design and run evaluations; reviews AI extraction outputs against ground truth; and delivers data-driven feedback to the team advancing the extraction process. The role also trains and supports business users on the ground-truth application, guides them through creating and reviewing ground-truth data, and collects source files and validated outputs. The analyst maintains a working understanding of the extraction processes and applies commercial risk insurance knowledge to define correct, complete extraction results.

Requirements

2–5+ years of experience in data quality, quality assurance, data analysis, annotation/labeling, or a related analytical or business-facing role, ideally with exposure to AI/ML, LLM, or data-extraction systems.
Experience contributing to structured testing, validation, or evaluation processes, such as QA test plans, user acceptance testing, or data or model validation.
Working knowledge of commercial risk insurance, including policy structure, coverage terms, limits, deductibles, exclusions, endorsements, submissions, and loss runs, sufficient to assess extraction accuracy and completeness.
General familiarity with large language models and generative AI, including their capabilities, limitations, and potential to produce inaccurate or fabricated outputs.
Strong analytical rigor and exceptional attention to detail, with the ability to assess extraction accuracy and turn findings into actionable recommendations.
Excellent communication and stakeholder-engagement skills, including the ability to train and support business users and build ground-truth workflows with non-technical partners.
Comfort working with data and data tools.
Hands-on AI/ML or software development experience is not required.

Nice To Haves

Bachelor’s degree in Computer Science, Information Systems, Mathematics, or a related field.
Familiarity with prompt engineering, retrieval-augmented generation (RAG), and LLM-as-judge evaluation.
Understanding of AI/ML and information-extraction evaluation methodology and metrics, such as precision, recall, F1 score, accuracy, and hallucination detection.
Experience building, curating, or maintaining labeled or ground-truth datasets and annotation workflows, including inter-annotator agreement.
Insurance industry designation such as CPCU, AINS, or ARM.
Basic proficiency in SQL for querying evaluation and production data.
Basic scripting or Python familiarity.
Familiarity with AI/LLM observability and evaluation tooling such as LangSmith.
Familiarity with Databricks and tools such as MLflow and Unity Catalog.

Responsibilities

Own, create, and maintain curated ground-truth datasets and labels used to measure, benchmark, and improve AI extraction quality.
Contribute to the design, execution, and continuous improvement of Enterprise AI Evaluation Strategies.
Help design evaluation plans and test sets using representative and edge-case samples across commercial risk insurance document types, lines of business, and levels of complexity.
Participate in baseline, regression, and pre-release evaluation cycles when models, prompts, or processes change, and help define production acceptance thresholds.
Help calibrate automated evaluation methods, including LLM-as-judge scoring, against human-reviewed ground truth.
Contribute to reusable enterprise evaluation standards, templates, and playbooks.
Define, document, and uphold annotation guidelines and standards for correct extraction.
Apply commercial risk insurance knowledge to judge extractions of policy and risk data and resolve ambiguous or complex cases.
Train and support business users on the ground-truth application, guide ground-truth creation and review, and collect source files and validated outputs.
Review AI extraction outputs against ground truth, categorize discrepancies and errors, and provide actionable feedback.
Coordinate human-in-the-loop review and annotation workflows, including inter-annotator agreement.
Manage ground-truth dataset versioning and lifecycle to keep quality results reproducible and comparable.
Partner with developers to define and interpret quality metrics and acceptance criteria, including accuracy, precision, recall, F1, completeness, groundedness/faithfulness, hallucination rate, and schema conformance.
Monitor extraction quality trends for drift, degradation, and anomalies, and escalate issues.
Maintain a working understanding of the team’s existing AI data extraction processes and partner with the Manager and team on continuous improvement.
Support responsible-AI review of outputs, including robustness, bias, safety, and compliance considerations.
Define and maintain quality metrics, dashboards, and reporting on extraction performance and ground-truth coverage.
Document ground-truth methodology, metric definitions, annotation standards, and quality practices.
Collaborate with Analytics Engineering and the Applied AI Analyst to align quality standards with downstream data and business needs.