Certification Assessment Specialist

OpenAI•San Francisco, CA
•$207,000 - $230,000

About The Position

The Certification Assessment Specialist designs, builds, tests, launches, and maintains hands-on assessments for an OpenAI technical role-based certification. The role turns competency models, assessment blueprints, and scoring rubrics into realistic candidate tasks, assessment environments, and workflows that capture evidence and produce results reviewers can verify. It uses scripting and automation and partners with engineering on software development and delivery integrations. This is a hands-on individual contributor role reporting to the Technical Certification Lead. The specialist collaborates with certification, measurement, engineering, and operations teams, technical subject matter experts, and external delivery providers. Assessments are taken from prototype through pilot, launch, and ongoing maintenance across live and virtual delivery. The specialist updates tasks, environments, and scoring as products and tools change, preserving alignment with the skills being assessed and keeping confidential assessment details separate from learning materials. Success means candidates can demonstrate the competencies required for certification and reviewers can consistently score their work using evidence traceable to the submission.

Requirements

  • Personally designed and built performance-based, hands-on assessments, labs, or simulations for technical roles such as software developers or solution architects, and taken them through testing, launch, and maintenance; question-and-answer tests alone do not meet this requirement.
  • Used scripting and automation to build and troubleshoot technical assessment environments.
  • Can work with technical subject matter experts to translate realistic solutions into assessments; deep AI-engineering expertise or experience with a particular language or delivery platform is not required.
  • Experience working at a technology company in a fast-paced environment, setting priorities, and delivering amid changing requirements and evolving business needs.
  • Can use a blueprint and rubric to design realistic tasks and specify evidence that distinguishes competent performance from task completion.
  • Built or maintained workflows that capture submissions, process evidence, and produce traceable scores or reviewer outputs.
  • Implemented or validated model-assisted scoring or evaluation against expert human judgments, and can explain when a score should be sent for human review.
  • Worked directly with external technical experts or delivery providers and can translate requirements into integrations that can be tested end to end.
  • Used pilot results, candidate feedback, and scoring or delivery data to improve assessment quality.
  • Understands how to build for accessibility and protect candidate evidence and confidential assessment content.
  • Can communicate technical tradeoffs clearly to subject matter experts, measurement specialists, platform engineers, and operations teams.

Nice To Haves

  • Experience building hands-on assessments for AI application development, agentic workflows, or customer deployments.
  • Experience working with psychometricians or measurement specialists on rubric calibration, scoring consistency, or validation of performance assessments.
  • Experience launching performance-based certification assessments across both live and remote delivery.

Responsibilities

  • Translate assessment blueprints into realistic scenarios, candidate instructions, and hands-on tasks; work with subject matter experts to ensure tasks reflect the job and elicit evidence appropriate to each rubric.
  • Define assessment environment requirements and build or configure candidate environments; establish repeatable provisioning and reset workflows for accounts, tools, data, and permissions.
  • Build and maintain pipelines that capture candidate submissions and scoring evidence, apply scoring rules, route results for review, ensure scores are traceable to evidence, and protect sensitive data under approved controls.
  • Build and test model-assisted scoring with qualified human reviewers; compare model scores with calibrated expert judgments, investigate disagreements, implement agreed routing rules for human review, and repeat validation after material changes.
  • Test assessments end to end, including environment setup, evidence capture, and scoring; use dry runs and pilots to evaluate instructions, timing, accessibility, and failure recovery; fix defects and verify fixes before release.
  • Test scoring against multiple valid approaches, partial and incorrect submissions, and assessment-environment failures so platform issues are not mistaken for candidate errors.
  • Translate assessment needs into delivery-integration requirements; partner with engineering and delivery providers to test candidate access, submission, and results handoffs and troubleshoot issues.
  • Prepare assessments for launch with platform and operations partners; document setup, version dependencies, and troubleshooting for consistent delivery and issue response.
  • Maintain versioned tasks, environments, scoring configurations, and test fixtures; use candidate feedback and delivery and scoring data to identify issues, update or retire tasks, and verify changes preserve intended skills and difficulty.
  • Investigate technical issues raised during delivery or candidate review; reproduce defects where possible, document their impact, and test fixes with relevant delivery partners.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service