Hybrid | Legal Counsel (AI Evaluation & Benchmarking) — $90–$140/hour

24-Mag • San Francisco, CA

1d • Hybrid

About The Position

We are sharing a specialised full-time opportunity for highly experienced attorneys with senior-level legal practice experience and the ability to evaluate, structure, and improve advanced AI systems working on complex professional legal tasks. This role works closely with AI research and programme teams to improve how advanced models reason about substantive legal work. Selected experts will review legal tasks and model outputs, develop instruction specifications and reference solutions, design challenging benchmarks, and translate senior-level legal judgement into clear, reproducible evaluation standards.

Requirements

• Juris Doctor (JD) from an accredited law school
• 5+ years of substantive post-qualification legal practice
• Professional experience within an established law firm, corporate legal department, regulatory body, court, or comparable legal institution
• Genuine specialisation in at least one substantive legal practice area
• Relevant expertise may include corporate and transactional law, litigation, regulatory and compliance, intellectual property, employment and labour, or tax
• Demonstrated progression to a senior legal position such as Partner, Of Counsel, Counsel, Senior Associate, Senior In-House Counsel, General Counsel, Chief Legal Officer, or comparable level
• Active admission to at least one U.S. state bar and good standing
• Direct ownership of significant legal matters, transactions, disputes, regulatory work, or strategic advisory responsibilities
• Hands-on professional experience using large language models or AI tools
• Strong ability to distinguish rigorous legal reasoning from plausible but technically incorrect analysis
• Excellent written communication and ability to provide precise, structured feedback
• Must live in the San Francisco Bay Area or be willing to relocate there before the engagement begins
• Must be able to work onsite with the assigned team multiple days per week when required

Responsibilities

• Review complex legal knowledge-work tasks and AI-generated outputs for substantive accuracy and professional quality
• Identify weak reasoning, missing issues, unsupported conclusions, and outputs that would not withstand senior legal scrutiny
• Evaluate legal analyses for completeness, precision, practical relevance, and appropriate application of legal principles
• Assess model performance across sophisticated legal scenarios
• Provide precise written feedback explaining errors, omissions, and required improvements
• Develop detailed instruction specifications for advanced legal AI tasks
• Produce high-quality reference solutions that establish clear standards for correct legal reasoning
• Design tasks that reflect authentic workflows encountered in senior legal practice
• Translate experienced legal judgement into explicit, teachable, and reproducible evaluation criteria
• Ensure reference materials demonstrate professional-level substantive depth and analytical rigour
• Design challenging legal benchmarks and evaluation sets
• Create tasks testing issue spotting, substantive knowledge, analytical reasoning, and professional judgement
• Develop criteria that distinguish rigorous legal analysis from plausible but incomplete responses
• Refine evaluation materials based on model performance and research requirements
• Help develop legal-specific AI capabilities, evaluation methodologies, and supporting tools
• Apply deep expertise from one or more substantive practice areas such as corporate and transactional law, litigation and dispute resolution, regulatory and compliance, intellectual property, employment and labour law, or tax
• Evaluate complex legal matters using professional methodologies and applicable legal frameworks
• Identify nuanced issues that may be overlooked by generalist reviewers
• Apply judgement developed through ownership of significant matters, transactions, disputes, investigations, or advisory work
• Maintain standards consistent with senior law-firm, in-house, regulatory, or comparable legal practice
• Collaborate with AI researchers, programme teams, and specialists across adjacent professional domains
• Participate in calibration processes to maintain consistent evaluation standards
• Apply hands-on experience with large language models to professional legal workflows
• Distinguish genuinely well-reasoned responses from outputs that are persuasive but substantively incorrect
• Communicate complex legal concepts clearly to technical and non-legal stakeholders

Benefits

• Full-time W-2 employment position
• Initial engagement of approximately 6 months
• 40 hours per week
• Hybrid working arrangement
• Regular onsite participation is required when requested
• Client-issued accounts and equipment will be provided as required

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: Ph.D. or professional degree