Remote | Humanities Evaluation Specialist — $40–$90/hour

24-Mag•New York, NY

2d•Remote

About The Position

We are sharing a specialised consulting opportunity for experienced humanities scholars, researchers, and professionals with strong expertise in interpretation, argumentation, source analysis, and analytical writing to contribute to an advanced AI evaluation project. Selected professionals will develop original, high-difficulty humanities questions, research and verify authoritative answers, and test whether advanced AI systems can reason effectively across interpretation, context, argument, and complex source material. The work requires strong analytical judgement, source triangulation, written precision, and the ability to create rigorous evaluation material that goes well beyond surface-level recall. No prior experience in AI is required.

Requirements

Advanced academic background or substantial professional experience in a humanities discipline
Relevant fields may include history, literature, philosophy, linguistics, classics, art history, journalism, or related areas
Strong analytical writing and close-reading skills
Demonstrated ability to formulate nuanced, challenging, and original questions
Experience researching and triangulating information from authoritative sources
Strong ability to document reasoning and citations clearly
Exceptional attention to detail and written precision
Self-directed and reliable when completing expert-level work independently
Comfortable working within remote, asynchronous project environments

Nice To Haves

Familiarity with AI model evaluation or training is advantageous but not required

Responsibilities

Develop original, high-difficulty question-and-answer pairs across humanities disciplines
Create prompts that test interpretation, argumentation, contextual understanding, and analytical reasoning
Design questions that require deep insight rather than straightforward factual recall
Ensure tasks reflect genuine disciplinary complexity and expert-level judgement
Maintain originality, intellectual depth, and methodological rigour across submitted work
Research and verify answers using primary texts, authoritative references, and reputable secondary sources
Provide clear citations supporting factual claims, interpretations, and analytical conclusions
Triangulate information across multiple sources where necessary
Distinguish well-supported interpretations from weak, ambiguous, or insufficiently evidenced claims
Document reasoning and source support clearly and consistently
Test developed questions against advanced AI models
Identify questions that are insufficiently challenging or susceptible to superficial answers
Iterate on prompts to increase difficulty while preserving accuracy and clarity
Analyse where AI systems succeed or fail in humanities reasoning and interpretation
Refine question structure to better measure genuine understanding and analytical capability
Write questions and answers with exceptional clarity and attention to detail
Eliminate ambiguity while preserving the intended intellectual difficulty
Produce well-supported answers that clearly explain the reasoning behind conclusions
Incorporate reviewer feedback and revise submissions where required
Follow project guidelines and maintain consistent quality standards across deliverables

Benefits

Independent contractor engagement
Fully remote
Compensation: $40–$90/hour
Compensation is output-based, with payment made for tasks that meet project specifications
Minimum weekly submission requirements apply
Work will involve humanities question development, source research, reference-answer authoring, AI model testing, and iterative refinement
Task completion time may vary depending on research complexity, discipline, and individual workflow
Selected professionals should be prepared to begin their first tasks within approximately 24–48 hours of completing onboarding
Roles are typically filled within approximately 48 hours
Project scope, workload, and evaluation standards may evolve depending on project requirements
Work must be completed without using confidential or proprietary information belonging to any employer, client, institution, research group, or other third party