Senior Machine Learning Engineer, AI Safety Evaluations

NVIDIA•Santa Clara, CA

About The Position

NVIDIA is seeking a machine learning engineer, AI researcher, or data scientist to support its AI Safety and Responsibility efforts within Enterprise Risk Management. The role applies machine learning, data science, and statistical skills to turn open research questions into rigorous benchmarks, experiments, and tools that assess how multimodal large language models behave under pressure. The engineer will work with researchers and platform engineers to connect evaluation findings to improved training methods and safeguards, producing reproducible results that inform technical reviews and executive decisions.

Requirements

Bachelor’s degree or equivalent experience.
At least 6 years of experience in machine learning engineering or evaluation.
Experience developing benchmarks, metrics, and statistically sound comparisons for machine learning systems, including selecting baselines and interpreting uncertainty.
Ability to translate ambiguous evaluation questions into testable experiments.
Strong Python skills and experience building maintainable shared tools, data pipelines, and experiment-tracking workflows.
Ability to assess dataset quality, identify limitations in metrics and scoring methods, and support conclusions with reproducible evidence.
Ability to clearly explain findings and tradeoffs to researchers, engineering partners, and executives.

Nice To Haves

Experience evaluating LLMs, multimodal models, or agentic systems through adversarial testing or automated red-teaming.
Contributions to public benchmarks or evaluation frameworks.
A publication track record and interest in continuing to publish at top-tier venues.
Experience validating model-based evaluators against human judgments or designing human-in-the-loop evaluation workflows.
Strong model evaluation skills and a commitment to sound statistical practices, including confidence intervals.

Responsibilities

Design and maintain safety benchmarks for large language and multimodal models by translating risk scenarios into test cases, datasets, and scoring rubrics.
Define evaluation metrics and protocols that assess safety alongside model usefulness, using baselines, uncertainty estimates, and failure analysis.
Build reusable Python pipelines with versioned datasets, model configurations, and experiment tracking to reproduce results and compare model releases.
Develop automated red-teaming tools to discover model failure modes and expand evaluation coverage, incorporating human review to validate automated scoring.
Investigate model failures with researchers, document actionable findings, and develop regression tests to assess whether model changes resolve identified issues.
Partner with platform engineers to store and aggregate evaluation results and communicate findings, limitations, and emerging risks to researchers and decision-makers.