Evaluation Data Scientist

COMMON SENSE MEDIA • San Francisco, CA

13d • $140,000 - $166,250 • Onsite

About The Position

Common Sense Media is the leading nonprofit organization dedicated to improving the lives of kids and families by providing the research-backed information, education, and independent voice they need to thrive in the age of apps, algorithms, and AI. We rate, educate, and advocate for policies to protect and prepare kids online. Our ratings, research, and resources reach more than 150 million users globally, over 1.4 million educators, and more than 100,000 schools worldwide every year. Learn more at commonsense.org. As part of Common Sense Media, the Youth AI Safety Institute sets standards, conducts research, and independently tests the AI products children use most. The Evaluation Data Scientist partners closely with the standards and evaluations team and owns the technical core of designing the evaluation methodology, establishing the statistical standards that make findings defensible, and building the infrastructure that keeps the Youth AI Safety Institute's science credible over time. Beyond the Institute, this role serves as a senior data science partner to the broader organization, partnering with teams across editorial, education, research, and advocacy to build KPI frameworks, design impact analyses, and translate program goals into measurable outcomes.

Requirements

• 6–8+ years of applied data science and ML experience, with meaningful recent focus on generative AI evaluation.
• Demonstrated command of the AI evaluation stack: LLM-as-judge design, synthetic persona generation, multi-turn testing, human-in-the-loop validation, and agreement statistics.
• Production fluency in Python, the modern ML/AI stack (transformers, LLM APIs, PyTorch/sklearn), and SQL.
• Experience designing evaluation instruments, ideally for youth-specific or sensitive-population risks.
• Ability to operate as a technical peer to senior engineers: code review, contributing to builds, shared architectural ownership.
• Experience applying causal inference and evaluation methods to measure real-world program or product outcomes.
• Strong written and verbal communication: able to defend methodology to academic reviewers and translate it for executive and advocacy audiences.
• Strong understanding of data governance, privacy, and responsible data practices, with particular sensitivity to data involving children and families.

Nice To Haves

• Peer-reviewed publication or equivalent research output in AI evaluation, youth wellbeing, or developmental science.
• Experience in mission-driven research organizations, academic settings, or publishing-culture environments.
• Experience with self-hosted, privacy-preserving LLM deployments for sensitive data.
• Experience overseeing external technical partners or multi-party evaluation programs.
• Deep working knowledge of child and adolescent development — cognitive, social, and emotional — across age bands, and fluency in the literature on youth-specific digital harms.
• Familiarity with product analytics platforms (Amplitude, Google Analytics) and modern data stack tooling (dbt, Airflow, or equivalents).

Responsibilities

• Translate Common Sense Media's safety standards and expert SME judgment into measurable, reproducible evaluation protocols — preserving the qualitative expertise the methodology rests on while making it auditable and scalable.
• Establish and defend the Institute's statistical standards: inter-rater reliability, agreement analysis, sample size and power calculations, and uncertainty quantification on AI-scored outputs.
• Design and maintain the Youth AI Safety Institute's evaluation architecture: simulated persona generation across developmental age bands, multi-turn interaction harnesses that reflect how children actually use AI products, LLM-as-judge ensembles with calibrated scoring rubrics, and human-in-the-loop validation that benchmarks automated scoring against expert human graders.
• Build the feedback loop that catches model drift and ensures Institute findings from one evaluation period remain comparable to the next as AI products evolve.
• Build internal tooling and dashboards that the Risk Assessment team and subject matter experts use directly in their review work, making scientific rigor an integrated part of their process.
• Author methodology findings in AI evaluation venues; contribute technical rigor to youth-focused research outputs.
• Represent the Institute's methods to academic reviewers, policymakers, and AI labs whose products are tested.
• Communicate findings to non-technical audiences, including executive leadership and government offices.
• Provide methodological oversight for external evaluation partners; review their approaches, ensure findings are coherent and comparable across products, flag validity concerns.
• Ensure external partner outputs remain grounded in Common Sense Media's developmental research and youth-specific risk frameworks; maintain methodological consistency across evolving products; establish scientific trust benchmarks for partners; and safeguard the Institute's reputation by owning the technical validity of flagship evaluation results.
• Design and execute A/B tests, randomized controlled trials, and quasi-experimental studies (difference-in-differences, regression discontinuity) across Common Sense Media's products and programs.
• Lead rigorous program evaluations measuring how Common Sense Media's work affects kids, families, and educators — applying the same causal inference standards used in Institute evaluations.
• Develop predictive and statistical models supporting organization-wide decisions, including audience understanding and member behavior.
• Partner with the data team on shared data infrastructure and methodology; contribute to the data models and pipelines that support org-wide reporting and analysis.
• Serve as a senior technical voice: advise on methodology, raise the technical bar, and help non-technical colleagues use data and AI tools effectively.

Benefits

• Competitive nonprofit compensation and comprehensive benefits.
• A collaborative, flexible work environment with meaningful leadership access and visibility.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed