Lead Assessment Developer

Outsmart • Los Angeles, CA

20h • $120,000 - $180,000 • Hybrid

About The Position

Outsmart is a mission-driven, early-stage startup reimagining higher education for the age of AI by creating an entirely new collegiate experience that expands access and opportunity. Backed by $38M from top venture firms including Khosla Ventures, Forerunner, DST Global Partners, and Lightspeed, the company is led by former Duolingo executives and a founding team that also includes Emmy Award winners. With a strong culture centered around high-impact, detail-oriented team members, Outsmart is already seeing significant traction, generating 1–5 million weekly organic views on social media. You'll manage the full assessment development lifecycle at Outsmart — from item authoring through field testing, standard setting, scoring, and reporting. You'll work closely with psychometricians, faculty, product managers, and engineers to build rigorous, scalable assessments. As an AI-forward organization, you'll also help shape how AI tools are responsibly woven into our development and QA workflows.

Requirements

• Master's degree (Ph.D. preferred) in educational measurement, assessment , or a related field
• 3+ years managing operational, high-stakes assessment lifecycles end-to-end
• Proven experience leading standard-setting studies and managing diverse SME panels
• Experience managing external vendors for field tests and pilot studies
• Working knowledge of validity, reliability, and fairness standards (e.g., AERA/APA/NCME)
• Familiarity with cognitive complexity frameworks (DOK, Bloom's)
• Strong communicator who can translate psychometric concepts for non-technical audiences

Nice To Haves

• Knowledge of CTT, IRT, CDM, DIF, AI-assisted item generation, or accreditation reviews

Responsibilities

• Manage the full assessment development lifecycle at Outsmart — from item authoring through field testing, standard setting, scoring, and reporting
• Work closely with psychometricians, faculty, product managers, and engineers to build rigorous, scalable assessments
• Help shape how AI tools are responsibly woven into our development and QA workflows
• End-to-end assessment development operations for assigned assessments
• SME network — recruitment, training, calibration, and quality
• Standard-setting studies — method selection, panel facilitation, and final reporting
• Vendor and external partner relationships, SLAs, and security protocols
• Field testing and pilot operations
• Scoring operations for constructed-response and performance-based assessments
• Project timelines, risk identification, and stakeholder communication
• Core assessment documentation — specs, blueprints, training materials, technical reports

Benefits

• Competitive salary and equity
• Healthcare for you and your dependents (medical, dental, and vision)
• Parental leave for any full-time employee who has been with us for at least 6 months (birthing, non-birthing, and adoptive parents)
• 15 PTO days per year, a winter break, and 12+ paid company holidays per year
• Remote work days
• Lunch provided in-office daily
• Working with super talented people who want to make a difference in the world

What This Job Offers

• Job Type: Full-time
• Career Level: Manager