Founding ML Engineer

SourceIn•San Francisco, CA

About The Position

The Founding ML Engineer will be the first dedicated machine learning hire at an early-stage AI startup building an AI monitoring and assurance platform for customer-facing AI agents. The platform evaluates customer conversations against regulations and business requirements to identify issues, reduce risk, and provide evidence of accountability. The engineer will work directly with the CTO, own the applied ML strategy, build production systems, and help establish the technical foundations and future team. This is a hands-on production engineering role, not a research-only position. The engineer will design compliance, hallucination, grounding, risk, escalation, and quality detectors; create evaluation frameworks and data processes; build customer rubric onboarding workflows; improve speech and conversation intelligence; and oversee inference economics, monitoring, and model reliability. The role also includes technical leadership, collaboration with customer QA and compliance teams, participation in enterprise security and AI governance reviews, mentoring, and recruiting.

Requirements

Staff- or Principal-level applied ML expertise, with ownership of production ML, NLP, or LLM systems and accountability for their performance.
Experience building evaluation pipelines, developing ground truth from inconsistent human labels, and making explicit decisions about acceptable error rates.
Early-stage startup experience shipping products amid ambiguity, broad ownership, and limited supporting infrastructure; stated as essential.
Strong model-selection judgment across prompting, retrieval, fine-tuning, distillation, classical classifiers, and simpler rules or regexes.
Hands-on Python engineering experience, including training or debugging models, building and maintaining production services, and working with messy, unstructured data.
Production ML judgment covering inference economics, latency, model reliability, sensitive data handling, and enterprise security requirements.
Ability to set direction for an ML function, establish high engineering standards, and recruit engineers.
Comfort using coding agents to accelerate development while maintaining rigorous testing, security, and maintainability standards.

Nice To Haves

Experience with production speech and audio systems, including ASR, diarisation, streaming transcription, or real-time latency optimisation.
Experience with custom NLP or scoring-model development, including LLM-as-judge systems validated against human ground truth.
Experience with weak supervision, programmatic labelling, or active learning in production.
Experience in trust and safety, fraud and risk, contact-centre quality assurance, regulatory compliance, or governance, risk and compliance (GRC).

Responsibilities

Design compliance, grounding, hallucination, risk, escalation, and interaction-quality detectors based on customer requirements and regulatory frameworks.
Select and combine rules, classifiers, LLM-based judges, and fallback mechanisms; establish confidence thresholds and promotion criteria that account for accuracy, latency, cost, and risk.
Own annotation guidelines, ground-truth datasets, adjudication processes, and offline and online evaluation.
Build regression gates to prevent ungrounded outputs, unsupported compliance claims, and unmeasured accuracy from reaching production.
Turn QA reviews, customer disputes, and human-versus-model disagreements into training and evaluation data.
Use synthetic data, weak supervision, and active learning where they demonstrably improve performance.
Build tools, prompts, datasets, and evaluation workflows to translate customer frameworks into reliable production detectors and speed up customer onboarding without compromising quality.
Develop reusable customer rubric approaches while accommodating customer-specific requirements.
Improve the reliability and scoring of noisy transcripts using NLP and LLM techniques, and evaluate transcription quality and its effect on downstream compliance and scoring.
Help design speech-processing capabilities, including ASR, speaker diarisation, and streaming transcription.
Manage inference cost and latency per interaction, model routing, and trade-offs between model capability and operational efficiency.
Monitor model drift and changes in compliance detection, grounding, hallucination detection, and scoring accuracy; build monitoring and alerting to surface regressions.
Partner with the CTO and design partners' QA and compliance teams to understand real-world model failures.
Contribute to enterprise security and AI governance reviews.
Establish ML engineering standards, evaluation practices, and technical direction for future hires.
Remain hands-on while mentoring engineers and building the team.