Head of AI and ML

Xplor•Remote, OR

5d•$275,000 - $290,000

About The Position

The Head of AI/ML R&D owns the scientific evaluation of AI systems across Xplor, determining whether systems work, how well they work, for whom, and at what cost. The role owns evaluation science across agents, predictive and decision models for shared agents—including churn, demand, payment recovery and fraud—and experiment design that enables Finance to measure AI value against a holdout. This is a hands-on role: at least 50% of the time is spent building models, evaluations and experiments, with the remainder devoted to setting methods, reviewing work and developing a small team. The role helps make shipped AI systems measurable, more cost-effective and trustworthy, and provides models for problems where an LLM is not the appropriate tool. The role also prototypes AI techniques against Xplor problems, converts successful approaches into reference implementations, monitors developments in models and methods, contributes statistical design and entity-resolution modeling for Data-as-a-Service, and raises ML fluency across the R&D organization. The position writes production code, notebooks and evaluation harnesses; builds initial evaluation suites and predictive models for golden-path examples; reviews model and evaluation work; and prioritizes demonstrations, evaluations and metrics over presentations.

Requirements

12+ years in data science, machine learning or applied research, including 4+ years leading teams of scientists or ML engineers.
Experience training, evaluating, deploying and maintaining production models, rather than only consuming APIs.
Experience shipping LLM or agentic systems to production within the last two years, including the evaluation suite that gated them.
Deep knowledge of experimental design and causal inference, including A/B tests, holdouts, uplift modelling and matched cohorts, and when each is valid.
Strong statistical skills, including sampling, confidence intervals, calibration and awareness of multiple-comparison traps.
Current experience building with AI-native tools, including agentic coding assistants, MCP and agent frameworks, with recent work to demonstrate.
Production-quality Python and SQL skills, and the ability to credibly review others' code.
Experience with SaaS or marketplace data at scale, including multi-tenant, messy and multi-product data.
Ability to explain model trade-offs and uncertainty to executives in business terms such as revenue, margin and risk.
Hands-on ability to write production code, notebooks and evaluation harnesses and build models, evaluations and experiments.
Evidence-first approach, communicating results with uncertainty measures.
Builder orientation, measuring success by customer-facing delivery.
Pragmatism in selecting the simplest model that meets the bar and declining research without a business-unit sponsor.
Ability to teach and grow ML judgment among engineers who do not report to the role.
Security- and privacy-minded handling of training data, prompts and tool output; protect card data, credentials and customer data.
Ability to partner with business units as customers and work within the co-build-with-two-BUs rule.

Nice To Haves

Experience with churn, retention, propensity or pricing models in subscription or SMB software.
Payments machine learning experience, such as recovery, dunning optimization, fraud or risk scoring.
Experience with fine-tuning, distillation or small-model work to reduce inference cost.
Experience designing and validating LLM-as-judge systems and red-teaming model behavior.
Experience with Snowflake or similar platforms and feature stores, and Datadog or OpenTelemetry for model monitoring.
Familiarity with NIST AI RMF, ISO/IEC 42001, the EU AI Act, and privacy rules for analytics across Australia, New Zealand, the United States, the United Kingdom and the European Union.
Published research or open-source work.

Responsibilities

Set enterprise evaluation methodology, including golden-set design, sampling, labelling, inter-rater agreement, LLM-as-judge validation, slice analysis and statistical thresholds.
Own quality signals in the minimum telemetry contract, including task success, groundedness, hallucination and escalation rates.
Rerun evaluations for each business unit and tenant on every agent-core upgrade and identify regressions before customers encounter them.
Build and run predictive and decision models for shared-agent needs, including churn and retention uplift, payment-recovery propensity and timing, demand and capacity forecasting, and later fraud and onboarding risk.
Select the appropriate approach for each task, such as classical ML, a small or fine-tuned model, retrieval or a frontier LLM.
Ship models to production on the golden path and monitor them for drift and feedback loops.
Build evidence for model routing based on accuracy, cost and latency per task.
Reduce cost per successful task through distillation, caching strategy, smaller models, and prompt and context optimization.
Prototype techniques such as agent memory, planning, multi-agent patterns and synthetic data against real Xplor problems; turn effective approaches into golden-path reference implementations and retire ineffective ones.
Provide a monthly view of changes in models and methods and their implications for the roadmap.
Shape statistical design for Data-as-a-Service benchmarks and indices, including cohort minimums, anonymization and comparability across businesses.
Lead entity-resolution modeling with the data team.
Teach evaluation, experimentation and probabilistic thinking through AI-P/SDLC training and office hours.
Pair with business-unit engineers on difficult model problems in lighthouse business units and review evaluation designs for R2 and R3 agents.
Write production code, notebooks and evaluation harnesses, using AI coding and agent tools daily at platform-contributor fluency.
Build the first evaluation suite and predictive model for each golden-path exemplar, including conversational analytics, booking agent, retention and churn.
Review model and evaluation pull requests from pods and business units.
Make decisions based on demos, evaluations and metrics rather than presentations.