Director of Quality

LILT•Boston, MA

About The Position

The Director of Quality owns LILT’s quality system end to end, including standards, audits, compliance, procedures, continuous improvement, and the roadmap and operation of Quality Agents. The role will help shift Quality from a primarily reactive, execution-led function to a proactive, data-driven partner for customers and internal teams, with quality built into AI-native workflows. It combines customer audits and quality-system operations with development and evaluation of AI agents. As the single Quality partner for Customer Success, Production, Research, and Product, the Director translates customer requirements into measurable acceptance criteria, effective controls, and improved models and workflows. The role advises customers on quality, cost, and speed trade-offs, including the appropriate use of automated, agent-verified, and human-verified workflows, with stronger assurance for high-risk and regulated content. The Director is a hands-on quality subject-matter expert and operator with experience building AI agents and improving LLM output, alongside a strong foundation in traditional quality management. The role reports to the VP of Production and leads the Quality team, including a customer-facing Quality Manager and two Quality Coordinators.

Requirements

Demonstrable, hands-on experience designing, building, and running LLM-based agents in production.
Scripting or data skills, such as Python or SQL, and experience turning quality data into actionable metrics, preventive controls, and measurable improvements.
Proven experience improving LLM output quality.
Significant experience running quality systems, including leading internal and external audits, compliance programs, and writing and maintaining processes and procedures.
Experience leading structured continuous improvement, including root cause analysis, corrective and preventive actions, and KPI reviews.
Experience managing and developing people.
Ability to translate customer quality requirements into acceptance criteria and governance, advise on quality, cost, and speed trade-offs, and communicate credibly with customers, executives, and technical teams.

Nice To Haves

Quality certifications such as Six Sigma.
Background in localization, translation, or language services.
Experience in customer-facing quality roles for enterprise accounts.
A track record of collaborating with research scientists and product managers to turn quality insights into product requirements and shipped improvements.

Responsibilities

Evaluate, design, and build reliable Quality Agents with well-defined tools and clear, structured instructions.
Use orchestration patterns to evolve single-agent systems into multi-agent systems.
Define agent evaluation criteria, test sets, and success metrics; run agents in production with monitoring and escalation paths.
Drive adoption of Quality Agents across teams and demonstrate measurable reductions in manual quality effort.
Partner with Research and Product on ADQ, LLM model improvements, AI Review, AI QA, and emerging AI workflows.
Translate quality findings into product requirements, evaluation criteria, and feedback loops that improve model and workflow output.
Represent Quality in roadmap discussions and help determine when human review, AI review, or both are appropriate.
Own the approach to internal and external audits, including ISO certification and customer audits.
Plan and conduct periodic internal audits, manage evidence collection, and track corrective and preventive actions through closure.
Standardize and maintain SOPs, workflows, and quality documentation as AI workflows evolve.
Ensure quality processes meet customer, contractual, and regulatory requirements.
Partner with Customer Success and Delivery to define proactive quality programs, success metrics, and governance for strategic accounts.
Translate customer quality requirements into pre-delivery acceptance criteria, validation methods, and risk-appropriate controls.
Advise customers on quality, cost, and speed trade-offs and the appropriate use of automated, agent-verified, human-verified, or combined workflows.
Represent LILT's Quality function in customer business reviews, audits, and escalations, using quality data and metrics to demonstrate fitness for purpose.
Lead structured continuous improvement across the Commercial business, including root cause analysis, corrective and preventive actions, KPI reviews, and retrospectives.
Own a proactive quality metrics framework that makes errors, PQs, rework, and trends visible, and use it to prioritize improvements and demonstrate outcomes.
Strengthen controls across review, delivery, and follow-through, including mandatory account-specific checks, intervention on critical findings, and clear escalation ownership.
Partner with Product and Engineering to automate checks and escalations, and verify corrective actions prevent recurrence.
Partner with Expert Supply and Operations to embed processes, guidance, and quality feedback that maintain a high-quality expert pipeline.
Use quality findings and recurring failure patterns to improve expert qualification, guidance, and ongoing quality management.
Lead and develop the Quality team, establish its operating cadence, and adapt how it works as Quality Agents reduce manual coordination.
Oversee customer-funded, embedded Quality Manager staffing and ensure those roles deliver the quality programs and initiatives customers purchased.
Develop playbooks for AI-driven quality management and keep staffing-as-a-service delivery aligned with LILT's quality standards.