Remote | Member of Technical Staff, Frontier AI — $600,000–$2,000,000/year

24-Mag•New York, NY

1d•Remote

About The Position

This full-time, fully remote opportunity is for an experienced technical professional working across AI research, machine-learning data systems, evaluation, and real-world model performance. The role owns research and evaluation initiatives from problem framing through data design, quality calibration, and signal validation, with the goal of producing high-quality, defensible research signal and measurable improvements in AI systems. The work combines evaluation design, ML-oriented data development, failure analysis, quality calibration, and collaboration with researchers, domain experts, and operational teams. The professional is expected to exercise sound judgment in ambiguous, high-impact environments, assess evidence and research quality, and help determine whether research directions should continue, change, pause, or stop based on findings. Priorities and evaluation frameworks may evolve with experimental findings and system performance. Work must not use confidential or proprietary information belonging to any employer, client, institution, or other third party. The opportunity is offered through 24-MAG LLC, which connects experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.

Requirements

Strong professional judgment regarding research signal quality and whether findings support broader conclusions.
Experience designing ML-oriented datasets, evaluation frameworks, annotation systems, rubrics, or quality-assurance processes.
Ability to translate complex and ambiguous real-world system behavior into structured research and evaluation opportunities.
Strong ownership mindset and comfort making decisions in uncertain or rapidly evolving environments.
Excellent written and verbal communication skills.
Ability to clearly explain technical trade-offs, limitations, evidence quality, and research findings.
Proven experience working directly with researchers, technical experts, or domain specialists during project calibration and iteration.
Systems-level understanding of model, agent, or AI-system performance.

Nice To Haves

Experience with reinforcement-learning environments, simulators, or feedback-driven training systems.
Experience improving agentic systems or AI systems operating within real-world workflows.
Prior work in applied research or production environments with direct impact on deployed systems.
Experience designing evaluations for complex or real-world tasks.
Familiarity with expert incentive design or high-stakes technical research programs.

Responsibilities

Own research and evaluation initiatives from problem framing through data design, quality calibration, and signal validation.
Define rigorous approaches for determining whether experimental results provide reliable and defensible research signal.
Analyze model and system failures to identify root causes, edge cases, and improvement opportunities.
Evaluate whether datasets, experiments, and conclusions meet appropriate quality thresholds, and act as a quality gate when signal, data integrity, or evidence is insufficient.
Design ML-oriented data systems, including task definitions, annotation schemas, rubrics, incentives, and supporting pipelines.
Structure data and evaluation workflows around downstream model-performance objectives.
Translate ambiguous real-world behavior into measurable evaluation frameworks and new data categories.
Identify evaluation or dataset coverage gaps and recommend areas for additional investment or iteration.
Develop quality-assurance processes that maintain consistent research standards.
Investigate model and system behavior to identify recurring weaknesses and performance limitations.
Iterate on evaluations, datasets, feedback loops, and quality standards.
Use experimental findings to guide improvements in model or agent performance.
Determine whether research directions should be expanded, revised, paused, or discontinued based on evidence.
Maintain a systems-level perspective focused on end-to-end AI performance rather than isolated components.
Collaborate with researchers, domain experts, operators, and cross-functional teams during project kickoff, calibration, and iteration.
Communicate findings, trade-offs, limitations, and signal strength to technical and non-technical stakeholders.
Translate research progress into credible, evidence-grounded narratives and support alignment between experimental work and real-world system requirements.
Contribute technical judgment in ambiguous, high-impact research environments.