Hybrid | Senior Legal AI Evaluation Counsel — $75–$110/hour

24-Mag • San Francisco, CA

1d • Hybrid

About The Position

We are sharing a specialised full-time opportunity for experienced counsel-level attorneys with substantial legal practice experience and the ability to evaluate, structure, and improve advanced AI systems working on complex professional legal tasks. This role works closely with AI research and programme teams to improve how advanced models reason about substantive legal work. Selected attorneys will review legal tasks and model outputs, develop instruction specifications and reference solutions, design challenging benchmarks, and translate experienced legal judgement into clear, reproducible evaluation standards.

Requirements

• Juris Doctor (JD) from an accredited law school
• Approximately 8–15 years of substantive post-qualification legal practice
• Professional experience within an established law firm, corporate legal department, regulatory body, court, or comparable legal institution
• Current or previous seniority at Counsel, Senior Associate, Senior In-House Counsel, Assistant General Counsel , or a comparable level
• Genuine specialisation in at least one substantive legal practice area
• Relevant expertise may include corporate and transactional law, litigation and dispute resolution, regulatory and compliance, intellectual property, employment and labour, or tax
• Active admission to at least one U.S. state bar and good standing
• Direct ownership of significant matters, transactions, disputes, regulatory work, or advisory responsibilities
• Hands-on professional experience using large language models or AI tools
• Strong ability to distinguish rigorous legal reasoning from plausible but technically incorrect analysis
• Excellent written communication and ability to provide precise, structured feedback
• Must live in the San Francisco Bay Area or be willing to relocate there before the engagement begins
• Must be able to work onsite with the assigned team multiple days per week when required

Responsibilities

• Review complex legal knowledge-work tasks and AI-generated outputs for substantive accuracy and professional quality
• Identify weak reasoning, overlooked issues, unsupported conclusions, and analyses that would not withstand professional legal scrutiny
• Evaluate legal work for completeness, precision, practical relevance, and appropriate application of legal principles
• Assess model performance across sophisticated legal scenarios
• Provide precise written feedback explaining errors, omissions, and required improvements
• Develop detailed instruction specifications for advanced legal AI tasks
• Produce high-quality reference solutions that establish clear standards for correct legal reasoning
• Design tasks reflecting authentic workflows encountered in professional legal practice
• Translate practitioner judgement into explicit, teachable, and reproducible evaluation criteria
• Ensure reference materials demonstrate appropriate substantive depth and analytical rigour
• Design challenging legal benchmarks and evaluation sets
• Create tasks testing issue spotting, substantive knowledge, analytical reasoning, and professional judgement
• Develop criteria that distinguish rigorous legal analysis from plausible but incomplete responses
• Refine evaluation materials based on model performance and research requirements
• Help develop legal-specific AI capabilities, evaluation methodologies, and supporting tools
• Apply specialist knowledge from one or more practice areas such as corporate and transactional law, litigation and dispute resolution, regulatory and compliance, intellectual property, employment and labour law, or tax
• Evaluate complex matters using professional methodologies and applicable legal frameworks
• Identify nuanced legal issues that may be overlooked by generalist reviewers
• Apply judgement developed through direct ownership of significant legal matters
• Maintain standards consistent with experienced counsel-level practice
• Collaborate with AI researchers, programme teams, and specialists across adjacent professional domains
• Participate in calibration processes to maintain consistent evaluation standards
• Apply hands-on experience with large language models to professional legal workflows
• Distinguish genuinely well-reasoned responses from outputs that are persuasive but substantively incorrect
• Communicate complex legal concepts clearly to technical and non-legal stakeholders

Benefits

• Full-time W-2 employment position
• Initial engagement of approximately 6 months
• 40 hours per week
• Hybrid working arrangement
• Regular onsite participation is required when requested
• Relocation assistance is not provided
• Client-issued accounts and equipment will be provided as required
• Work will be conducted within assigned internal tools and workflows

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: Ph.D. or professional degree