Member of Technical Staff, Pretraining Evaluations

Hark • San Jose, CA

1d • $180,000 - $450,000

About The Position

Hark is an artificial intelligence company developing personalized, multimodal intelligence and next-generation AI hardware. The Omni team builds AI experiences beyond text, with models that understand and generate content across modalities including text, audio, and more.

As a member of the Omni team, you will build multimodal foundation models and the systems used to evaluate Hark’s models during pretraining and mid-training. You will track held-out loss, measure capabilities across text and multimodal tasks, and make comparisons between training runs reproducible. Your analysis will help researchers choose training recipes, select checkpoints, and identify regressions.

Requirements

• Experience building model evaluation systems or experimental tools for reproducible comparisons across models or checkpoints.
• Knowledge of language model evaluation methods, including log-likelihood and generative scoring, prompt sensitivity, few-shot variance, and formatting effects.
• Strong statistical judgment, including uncertainty estimation, sample-size planning, and distinguishing statistical significance from practical improvements.
• Experience identifying benchmark contamination, saturation, or gaps between benchmark scores and the intended capability.
• Strong Python engineering skills, including building reliable evaluation pipelines that run efficiently across many checkpoints.
• A record of developing useful benchmarks, evaluation methods, or tools that inform research and engineering decisions.

Nice To Haves

• Experience designing benchmarks with human ratings or model-based graders.
• Experience studying capability scaling or predicting larger-model performance from smaller experiments.
• Experience with code execution sandboxes, math verifiers, or other programmatic graders.
• Experience evaluating checkpoints during pretraining or mid-training.
• Multimodal evaluation experience: audio, speech, vision, or interleaved inputs.
• Experience evaluating model quality under constrained inference budgets.

Responsibilities

• Build and maintain an evaluation harness for held-out loss, performance across data domains, and capabilities in text, code, math, and reasoning.
• Design evaluations for early checkpoints and smaller models using log-likelihood scoring, few-shot prompts, and generated completions.
• Assess benchmark coverage and saturation, quantify uncertainty in results, and work with data researchers to detect training-data contamination.
• Version datasets, prompts, graders, and inference settings so results can be reproduced and compared across checkpoints and training runs.
• Analyze experiments and work with researchers to identify useful recipe changes, select checkpoints, and prioritize capability improvements.
• Develop new benchmarks for capabilities that existing evaluations do not adequately measure.
• Extend evaluation tools to image, video, speech, and audio tasks in collaboration with multimodal researchers.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed