AI Quality Lead -Virtual

Alight

6h • Remote

About The Position

As the AI Quality Lead for the Artificial Intelligence Center of Excellence (AI CoE), you will architect and drive the end-to-end AI Quality program from inception. This is a hands-on leadership role, requiring both strategic vision and technical depth to go from 0→1 in AI Quality at Alight.

Requirements

• Experience leading or building quality engineering or testing programs (AI/ML experience is a plus, but not required)
• Ability to design and implement test strategies, automation frameworks, and quality standards
• Excellent communication and leadership skills; ability to work cross-functionally and influence stakeholders.
• Experience with LLM, RAG, agentic, or classic ML systems
• Familiarity with test automation frameworks, CI/CD pipelines, and evaluation tools (promptfoo, TruLens, Ragas)
• Knowledge of fairness, explainability, safety, and security standards (e.g., OWASP Top 10, MITRE ATLAS, Fairlearn, AIF360)
• Experience with data integrity, lineage, and observability (OpenTelemetry, New Relic, OpenLineage)
• Familiarity with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
• Knowledge of international standards and regulatory frameworks for Responsible AI
• Bachelor’s or Master’s degree in Computer Science, Engineering, Data Science, or related field, or equivalent experience

Responsibilities

• Design and implement the unified AI Quality strategy and SOP, scaling controls by risk tier (Tier 1–3)
• Define and own success metrics, risk tiering, acceptance criteria, and operating thresholds for AI solutions
• Lead the development of golden/adversarial test suites (prompt injection, misuse, safety, bias, functional automation)
• Oversee the creation and maintenance of schema-level UI/API tests, agent contracts, and observability instrumentation
• Own the org-wide evaluation harnesses (e.g., promptfoo, TruLens, Ragas), CI gating, and red-team playbooks
• Manage independent spot checks, factsheets, registry entries, and evidence for governance and audit
• Collaborate with Product Owners, Developers, SRE/Observability, and Governance to ensure compliance and operational excellence
• Drive continuous improvement in fairness, explainability, safety, security, and operational SLOs
• Mentor and grow the AI Quality team, including onboarding and coaching junior engineers
• Experience with LLM, RAG, agentic, or classic ML systems
• Familiarity with test automation frameworks, CI/CD pipelines, and evaluation tools (promptfoo, TruLens, Ragas)
• Knowledge of fairness, explainability, safety, and security standards (e.g., OWASP Top 10, MITRE ATLAS, Fairlearn, AIF360)
• Experience with data integrity, lineage, and observability (OpenTelemetry, New Relic, OpenLineage)
• Familiarity with AI infrastructure, orchestration, data integrity, deployments, and model registry practices
• Knowledge of international standards and regulatory frameworks for Responsible AI

Benefits

• We offer programs and plans for a healthy mind, body, wallet and life because it’s important our benefits care for the whole person. Options include a variety of health coverage options, wellbeing and support programs, retirement, vacation and sick leave, maternity, paternity & adoption leave, continuing education and training as well as a number of voluntary benefit options.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level