Hybrid | Attorney (AI Evaluation & Benchmarking) — $40–$80/hour

24-Mag • New York, NY

1d • Hybrid

About The Position

We are sharing a specialised full-time opportunity for experienced attorneys with deep expertise in a substantive legal practice area and the ability to evaluate, structure, and improve advanced AI systems working on real-world legal tasks. This role works closely with AI research and programme teams to improve how advanced models reason about professional legal work. Selected experts will review legal tasks and model outputs, develop instruction specifications and reference solutions, design challenging benchmarks, and translate senior-level legal judgement into clear, reproducible evaluation standards.

Requirements

• Juris Doctor (JD) from an accredited law school
• 5+ years of substantive post-qualification legal practice
• Professional experience within an established law firm, corporate legal department, regulatory body, court, or comparable legal environment
• Genuine specialisation in at least one substantive legal practice area
• Relevant expertise may include corporate and transactional law, litigation, regulatory and compliance, intellectual property, employment and labour, or tax
• Demonstrated progression to a senior legal position such as Partner, Of Counsel, Counsel, Senior Associate, Senior In-House Counsel, General Counsel, or comparable level
• Active admission to at least one U.S. state bar and good standing
• Direct ownership of significant legal matters, transactions, disputes, regulatory work, or advisory responsibilities
• Hands-on professional experience using large language models or AI tools
• Strong ability to distinguish rigorous legal reasoning from plausible but technically incorrect analysis
• Excellent written communication and ability to provide precise, structured feedback
• Must live in the San Francisco Bay Area and be able to attend occasional onsite sessions when required

Responsibilities

• Review legal knowledge-work tasks and AI-generated outputs for substantive accuracy and professional quality
• Identify weak reasoning, missing considerations, unsupported conclusions, and answers that would not withstand professional legal scrutiny
• Evaluate legal analyses for completeness, precision, and appropriate application of governing authority
• Assess model performance across complex legal scenarios
• Provide precise written feedback explaining errors and required improvements
• Develop detailed instruction specifications for legal AI tasks
• Produce high-quality reference solutions that establish clear standards for correct responses
• Design tasks that accurately reflect how legal work is performed in professional practice
• Translate experienced legal judgement into explicit and teachable evaluation criteria
• Ensure reference materials demonstrate strong substantive depth and practical legal reasoning
• Design challenging legal benchmarks and evaluation sets
• Create tasks that test substantive knowledge, analytical depth, issue spotting, and professional judgement
• Develop evaluation criteria that distinguish rigorous legal reasoning from plausible but incomplete responses
• Refine benchmarks based on model performance and research requirements
• Help develop legal-specific evaluation capabilities, workflows, and tools
• Apply specialist knowledge from areas such as corporate and transactional law, litigation and dispute resolution, regulatory and compliance, intellectual property, employment and labour law, or tax
• Evaluate complex legal problems using professional methodologies and applicable legal frameworks
• Identify nuanced issues that may be missed by generalist reviewers
• Contribute senior-level judgement grounded in substantive legal practice
• Maintain high standards of analytical rigour across legal evaluation workflows
• Collaborate with AI researchers, programme teams, and specialists across adjacent professional domains
• Participate in calibration processes to maintain consistent evaluation standards
• Apply hands-on experience with large language models to professional legal workflows
• Distinguish genuinely well-reasoned outputs from responses that sound persuasive but are legally incorrect
• Communicate complex legal concepts clearly to technical and non-legal stakeholders

Benefits

• Full-time W-2 employment position
• Initial engagement of approximately 6 months
• 40 hours per week
• Hybrid working arrangement, with day-to-day work performed remotely and occasional onsite collaboration in the San Francisco Bay Area
• Client-issued accounts and equipment will be provided as required for the role
• Work will be conducted within assigned internal tools and workflows

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: Ph.D. or professional degree