Director, AI & Tooling

Salesforce•Boston, MA

5d•$164,000 - $304,700

About The Position

The Director of Help Agent AI & Tooling leads the team accountable for the generative AI capabilities, evaluation tooling, observability, and emerging program initiatives that power Salesforce's Help Agent — a customer-facing Agentforce deployment and one of Salesforce's flagship Customer Zero implementations. This leader sits at the intersection of AI product strategy, data-driven evaluation, and production excellence, and plays a critical role in ensuring the Help Agent performs reliably, improves continuously, and delivers measurable business value.

Requirements

Generative AI capabilities
Evaluation tooling
Observability
Emerging program initiatives
AI product strategy
Data-driven evaluation
Production excellence
Retrieval architecture (e.g., Ensemble Retriever)
LLM configurations
Prompt and planner optimization
Agentic readiness scoring
Testing Center
Agentforce Testing Agents (ATA)
Custom evaluation frameworks
Answer quality metrics (completeness, correctness, relevance, faithfulness, clarity, and latency)
Real-time performance monitoring
Session tracing
Error analysis
Failure clustering
Business impact measurement
Net New Annual Order Value (NNAOV) program management
AI Skill & Application Development
Slackbot
Claude Code
Team mission, operating model, and priorities
Customer Zero stewardship
Shiproom intakes
Direct T&P partnerships
Help Agent production health (end-to-end latency, engagement rate, escalation rate, resolution rate, and quality scores)
Cross-functional collaboration with Technology Teams
AI Evaluations Framework
Scalable, automated evaluation at production scale
Observability roadmap
Centralized, production-grade monitoring platform
Leadership forums communication

Nice To Haves

Salesforce's own use of Agentforce as a reference customer
Surfacing product gaps
Driving roadmap influence
Replacing fragmented tooling
Center of excellence for generative AI evaluation and tooling practices across Digital Success and the broader Salesforce Customer Zero ecosystem

Responsibilities

Drive the strategic roadmap for generative AI capabilities within the Help Agent — including retrieval architecture (e.g., Ensemble Retriever), LLM configurations, prompt and planner optimization, and agentic readiness scoring. Partner closely with Tech & Product, and other technology teams to influence the Help Agent roadmap and direct engineering relationships.
Own the end-to-end evaluations tooling strategy for Help Agent performance — including the Testing Center, Agentforce Testing Agents (ATA), and custom evaluation frameworks. Define and track answer quality metrics including completeness, correctness, relevance, faithfulness, clarity, and latency. Manage tooling gaps, coordinate with Tech & Product on product priorities, and expand evaluation capabilities toward greater scale and automation.
Lead the observability program for the Help Agent — encompassing real-time performance monitoring, session tracing, error analysis, failure clustering, and business impact measurement. Serve as the primary stakeholder and voice of the Customer Zero on the Tech & Product Observability product roadmap. Coordinate with Technology Teams to converge on a comprehensive, always-on observability solution that replaces fragmented tooling.
Oversee the Net New Annual Order Value (NNAOV) program, driving pipeline generation through the Help Agent. Partner with the dedicated NNAOV Program Manager to define program goals, manage intake, and report on business outcomes.
Support the creation and delivery of AI Skills via applications like Slackbot, Claude Code and others to support the team's ability to deliver evaluations, insights, and opportunity sourcing for Roadmap development.
Serve as the AI & Tooling team lead within Digital Success Data & AI, setting team mission, operating model, and priorities
Act as a Customer Zero steward — representing Salesforce's own use of Agentforce as a reference customer, surfacing product gaps and driving roadmap influence through Shiproom intakes and direct T&P partnerships
Monitor and manage Help Agent production health, including end-to-end latency, engagement rate, escalation rate, resolution rate, and quality scores — escalating issues and coordinating cross-functional remediation as needed
Lead cross-functional collaboration with Technology Teams to align on evaluations, retrieval strategies, and tooling development
Define and mature the AI Evaluations Framework, moving from manual, spreadsheet-based processes toward scalable, automated evaluation at production scale
Drive the observability roadmap in partnership with Technology Teams, from current OOTB capabilities to a fully centralized, production-grade monitoring platform
Represent AI & Tooling in leadership forums and communicate progress, risks, and strategic direction clearly to senior stakeholders