AI Quality Lead -Virtual

Alight Solutions

20d • $125,000 - $180,000 • Remote

About The Position

As the AI Quality Lead for the Artificial Intelligence Center of Excellence (AI CoE), you will architect and drive the end-to-end AI Quality program from inception. This is a hands-on leadership role, requiring both strategic vision and technical depth to go from 0→1 in AI Quality at Alight.

Requirements

• Experience leading or building quality engineering or testing programs (AI/ML experience is a plus, but not required)
• Ability to design and implement test strategies, automation frameworks, and quality standards
• Excellent communication and leadership skills; ability to work cross-functionally and influence stakeholders.
• Experience with LLM, RAG, agentic, or classic ML systems
• Familiarity with test automation frameworks, CI/CD pipelines, and evaluation tools (promptfoo, TruLens, Ragas)
• Knowledge of fairness, explainability, safety, and security standards (e.g., OWASP Top 10, MITRE ATLAS, Fairlearn, AIF360)
• Experience with data integrity, lineage, and observability (OpenTelemetry, New Relic, OpenLineage)
• Familiarity with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
• Knowledge of international standards and regulatory frameworks for Responsible AI
• Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or related field, or equivalent experience
• “This role supports services provided to the federal government or federal government contractor and therefore requires US Citizenship. Proof of citizenship status will be required at time of hire.”

Nice To Haves

• Experience with LLM, RAG, agentic, or classic ML systems
• Familiarity with test automation frameworks, CI/CD pipelines, and evaluation tools (promptfoo, TruLens, Ragas)
• Knowledge of fairness, explainability, safety, and security standards (e.g., OWASP Top 10, MITRE ATLAS, Fairlearn, AIF360)
• Experience with data integrity, lineage, and observability (OpenTelemetry, New Relic, OpenLineage)
• Familiarity with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
• Knowledge of international standards and regulatory frameworks for Responsible AI

Responsibilities

• Design and implement the unified AI Quality strategy and SOP, scaling controls by risk tier (Tier 1–3)
• Define and own success metrics, risk tiering, acceptance criteria, and operating thresholds for AI solutions
• Lead the development of golden/adversarial test suites (prompt injection, misuse, safety, bias, functional automation)
• Oversee the creation and maintenance of schema-level UI/API tests, agent contracts, and observability instrumentation
• Own the org-wide evaluation harnesses (e.g., promptfoo, TruLens, Ragas), CI gating, and red-team playbooks
• Manage independent spot checks, factsheets, registry entries, and evidence for governance and audit
• Collaborate with Product Owners, Developers, SRE/Observability, and Governance to ensure compliance and operational excellence
• Drive continuous improvement in fairness, explainability, safety, security, and operational SLOs
• Mentor and grow the AI Quality team, including onboarding and coaching junior engineers

Benefits

• We offer programs and plans for a healthy mind, body, wallet and life because it’s important our benefits care for the whole person. Options include a variety of health coverage options, wellbeing and support programs, retirement, vacation and sick leave, maternity, paternity & adoption leave, continuing education and training as well as a number of voluntary benefit options.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level