Data Science - AI

Ironclad•San Francisco, CA

1d•Hybrid

About The Position

Ironclad is accelerating its investment in AI to redefine how legal teams manage and understand contracts. As part of this effort, we are hiring an AI Evaluation Engineer to work within our AI Pillar. This role is focused on unlocking insights from our training data, designing feedback loops, and ensuring the continuous improvement of our agentic and ML or LLM-based systems through data-driven evaluation and iteration. You’ll partner closely with AI Engineers and Product Managers to drive better model quality through systematic analysis, experimentation, and the curation of high-leverage datasets. Your work will directly impact the effectiveness of features like Smart Import, contract understanding, and agentic workflows.

Requirements

Bachelor's or Master's degree in a quantitative field (e.g., Statistics, Computer Science, Data Science, Applied Math).
1–3 years of experience in applied ML or data science, preferably in NLP or LLM-based applications.
Strong SQL and Python skills; experience with Jupyter, Pandas, and experiment tracking tools.
Comfortable navigating ambiguity, slicing large datasets, and communicating insights clearly to cross-functional stakeholders.

Nice To Haves

Experience with prompt analysis, clustering, or user behavior modeling is a plus.
Familiarity with LLM eval techniques, Reinforcement Learning from Human Feedback (RLHF), or agentic system design.
Experience with program management.

Responsibilities

Analyze training and evaluation datasets to identify distributional gaps, labeling inconsistencies, and long-tail opportunities.
Design and execute labeling campaigns, including development of golden datasets and annotation guidelines.
Build and maintain dashboards that track model accuracy, regression trends, and product-specific KPIs like success rate or answer helpfulness.
Investigate failure modes via prompt clustering, error taxonomy development, and user intent classification.
Operationalize feedback loops: mine product telemetry and human-in-the-loop reviews for signal, then translate into data-driven model improvement strategies.
Partner with engineers and PMs to run structured A/B tests and human evaluations for new models or features.
Support the development of scalable data and evaluation infrastructure for LLMs and agents.
Work with product, engineering and legal to create clear & transparent processes for the handling of customer data in AI training, fine-tuning and evaluation

Benefits

100% health coverage for employees (medical, dental, and vision), and 75% coverage for dependents with buy-up plan options available
Market-leading leave policies, including gender-neutral parental leave and compassionate leave
Family forming support through Maven for you and your partner
Paid time off - take the time you need, when you need it
Monthly stipends for wellbeing, hybrid work, and (if applicable) cell phone use
Mental health support through Modern Health, including therapy, coaching, and digital tools
Pre-tax commuter benefits (US Employees)
401(k) plan with Fidelity with employer match (US Employees)
Regular team events to connect, recharge, and have fun
The opportunity to help build the company you want to work at