Remote | Machine Learning & NLP Research Evaluator — $75–$105/hour

24-Mag•New York, NY

14h•Remote

About The Position

We are sharing a specialised part-time consulting opportunity for machine learning and natural language processing professionals experienced in model development, language understanding, retrieval systems, transformer architectures, Python workflows, and advanced AI evaluation. This role supports a remote collaboration focused on improving how advanced AI systems reason across machine learning and natural language processing tasks. Selected professionals will apply their technical expertise to design realistic ML and NLP problems, develop rigorous reference solutions, evaluate model behavior, identify capability gaps, and contribute to high-quality technical assessment workflows.

Requirements

Deep hands-on experience in machine learning or natural language processing through industry work, academic research, graduate study, or open-source contributions
Practical proficiency in Python used in research, production, or technical development environments
Strong understanding of modern ML and NLP methods, including model training, evaluation, and transformer-based systems
Experience with large language models, retrieval systems, language generation, or applied machine learning pipelines
Ability to design technically rigorous tasks and analyze model behavior independently
Strong written communication and time-management skills
Availability to contribute approximately 20 hours per week
Current location in the United States
A degree in computer science, machine learning, artificial intelligence, computational linguistics, data science, mathematics, or a related technical field is helpful
Graduate-level research or doctoral training in machine learning, NLP, artificial intelligence, or a related discipline is highly relevant
Professional experience developing or evaluating production ML or NLP systems is also highly valuable
Equivalent hands-on experience through established open-source, research, or applied technical contributions may be considered

Nice To Haves

Experience with PyTorch, TensorFlow, JAX, Hugging Face, or comparable machine learning frameworks
Familiarity with transformers, large language models, embeddings, retrieval-augmented generation, or sequence modeling
Experience building datasets, benchmarks, evaluation harnesses, or reproducible ML pipelines
Knowledge of standard machine learning and NLP metrics, error-analysis methods, and validation practices
Experience with information retrieval, text classification, semantic search, summarization, or language generation
Previous involvement in AI training, model evaluation, technical benchmarking, or structured data review
Open-source contributions related to machine learning, NLP, evaluation tooling, or model infrastructure

Responsibilities

Design challenging, real-world machine learning and NLP problems based on applied industry, research, or open-source experience
Develop tasks involving model training, language understanding, text generation, retrieval, classification, ranking, and applied ML pipelines
Create problems that target specific technical weaknesses in advanced AI systems
Calibrate task difficulty, expected behavior, and evaluation criteria against modern ML and NLP standards
Develop clear specifications, reference solutions, and supporting technical materials
Prepare executable tests, validation logic, and evaluation components using Python where applicable
Integrate tasks into structured development and assessment environments
Review reference implementations for correctness, reproducibility, and technical quality
Evaluate model outputs across machine learning, natural language processing, retrieval, generation, and agentic workflows
Assess outputs for correctness, robustness, consistency, and alignment with task requirements
Identify failures involving reasoning, implementation, data handling, model behavior, or language interpretation
Document evaluation decisions clearly and support findings with technical evidence
Identify tasks where meaningful model-performance headroom remains
Classify technical failures according to their underlying cause, severity, and impact
Distinguish between reasoning errors, modeling limitations, implementation issues, and evaluation constraints
Collaborate with other subject-matter experts to maintain consistent and accurate assessment standards