AI Engineering Lead, Evals and Systems Integration

Bain & Company • Boston, MA

2d • $189,500 - $237,500

About The Position

The AI Engineering Lead, Evals and Systems Integration owns the evaluation framework and systems integration layer across Bain’s Growth AI Performance System platform. This cross-cutting role assesses the coherence, quality, and reliability of the full system, from the data ontology layer through the Agentic Core to three product modules, and establishes the quality gate for determining deployment readiness.

The role works across the Data Ontology and Knowledge team, the Agentic Core and Interface team, and the Platform and Security team. It evaluates semantic-layer outputs, agentic workflows, integration points, and the security and compliance of evaluation infrastructure. Across modules focused on marketing ROI, brand, and pricing strategy, the lead defines domain-specific quality standards, evaluation datasets, rubrics, and deployment-readiness requirements.

The position has significant client proximity. The lead engages with consulting teams after deployments to understand system shortcomings, turns feedback into structured evaluation improvements, and presents quality and reliability findings to senior client stakeholders. The role combines production AI engineering, cross-team coordination, and evidence-based communication about what the platform can be trusted to do.

Requirements

• Deep experience designing and running evaluations for LLM-based and agentic AI systems across the delivery lifecycle, with evidence that evaluation frameworks influence what ships.
• Strong understanding of systems integration evaluation, including interfaces among knowledge graph outputs, API contracts, and agentic context passing.
• Strong Python proficiency and experience building automated evaluation pipelines, regression suites, and scoring rubrics integrated with CI/CD.
• Ability to apply statistical reasoning and experimental design to measure system quality, interpret results, and identify root causes of failures.
• Familiarity with knowledge graph and ontology evaluation, including entity resolution correctness, taxonomy consistency, and semantic-layer quality.
• Experience with human-in-the-loop annotation and feedback systems; familiarity with preference learning, data labeling, and feedback collection for LLM and agentic applications.
• Understanding of security, audit, and compliance requirements for evaluation infrastructure, including sensitive client-data handling and audit trails.
• Active use of AI-assisted coding tools such as Cursor, Claude Code, or equivalent, and ongoing development of evaluation methodology.
• Demonstrated commitment to responsible AI, including risk assessment, bias and fairness evaluation, AI safety, and awareness of prompt injection, data leakage, and agentic workflow exploitation.
• At least 8 years in AI engineering, ML engineering, or applied AI roles with significant hands-on building responsibilities.
• Track record of building and owning evaluation frameworks for production AI or ML systems, including defining standards, building infrastructure, and enforcing deployment gates.
• Experience evaluating multi-component systems and integration points across data pipelines, APIs, and agentic workflows.
• Direct experience engaging with clients or senior business stakeholders in a technical capacity.
• Experience in agile, cross-functional product or consulting-adjacent environments, with strong troubleshooting and root-cause analysis skills.
• Associate’s degree or bachelor’s degree in computer science, engineering, mathematics, or a related technical field, or an equivalent combination of education, training, and experience.

Nice To Haves

• Experience working with or alongside knowledge graph, ontology, or semantic data teams.

Responsibilities

• Own and evolve the evaluation framework across the Growth AI Performance System, defining quality standards, scoring rubrics, evaluation datasets, and deployment-readiness thresholds for the ontology layer, Agentic Core, and product modules.
• Build and maintain evaluation datasets and regression suites to measure quality across the stack, including knowledge graph accuracy, entity resolution, workflow reliability, and end-to-end output quality.
• Define and enforce deployment-readiness standards for the semantic layer, agentic orchestration, retrieval pipelines, and product outputs.
• Design experiments to measure quality, detect regressions, identify root causes, and distinguish meaningful signals from noise.
• Build automated evaluation pipelines integrated into CI/CD so quality gates run before deployments.
• Evaluate integration between the data ontology layer and Agentic Core, including knowledge graph structure, semantic consistency, and compatibility with retrieval and reasoning systems.
• Partner with the Data Ontology and Knowledge team on entity resolution, taxonomy consistency, and data contract evaluation criteria, tests, and monitoring.
• Partner with the Agentic Core and Interface team to evaluate multi-agent orchestration, tool use, context passing, workflow correctness, failure modes, and graceful degradation.
• Partner with the Platform and Security team to integrate evaluation infrastructure with security controls, audit logging, compliance requirements, and sensitive-data handling.
• Coordinate evaluation across distributed engineering teams, convert findings into prioritized engineering improvements, and communicate quality status to product and engineering leadership.
• Engage with consulting teams after client deployments, investigate shortcomings, trace failures, identify root causes, and feed structured findings to engineering teams.
• Present evaluation findings, quality metrics, and reliability assessments to senior stakeholders, translating technical measures into business terms.
• Contribute to internal communications explaining the platform’s evaluation approach and quality standards.
• Stay current with evaluation methodologies, responsible AI assessment frameworks, and AI risk and safety approaches, and lead knowledge sharing on best practices.

Benefits

• Annual discretionary performance bonus.
• 401(k) plan with an annual employer contribution based on years of service; the contribution is 4.5% and increases after 3 years of service, and is fully vested upon the start date.
• Medical, dental, and vision programs with Bain paying 100% of individual employee premiums.
• Paid time off, including parental leave, sick leave, and paid holidays.
• Paid life and long-term disability insurance.
• Potential eligibility for other elements of discretionary compensation.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: Associate degree