Core Speech AI Evaluation Engineer
About The Position
The Core Speech AI Evaluation Engineer will lead efforts to establish quality benchmarks and processes for AI-produced medical notes, including automated LLM-as-judge evaluation. The role implements data analysis pipelines to produce statistical quality reports for research and development and support teams, and contributes actionable analytics on the company’s NLP and NLU products. The engineer works with global teams on products supporting multiple languages, including English, German, Spanish, and Italian, and participates in a business-hours-restricted on-call rotation to resolve team tool errors. The role is listed as remote/hybrid and may include up to 5% domestic or international travel. Relocation assistance may be authorized. New employees whose start date is October 1, 2025 or later must travel to a designated company location for initial on-site onboarding; the company coordinates and pays related travel expenses under its travel policy.
Requirements
- Bachelor’s degree or higher and 4 years of data analytics experience, or high school diploma/GED and 8 years of data analytics experience.
- Experience in AI-model quality analysis.
- Must be legally authorized to work in the country of employment without employment-visa sponsorship.
Nice To Haves
- Experience in linguistics and/or human-computer interaction.
- Experience with natural language processing/understanding.
- Experience with medical/clinical software development.
- Experience in evaluation design, including rater training, blinding, inter-rater agreement, sample sizing, and identifying noise.
- Specific experience in AI-model quality and safety analysis.
- Skill in working across team boundaries toward a shared goal.
Responsibilities
- Lead efforts to establish quality benchmarks and processes for AI-produced medical notes, including automated LLM-as-judge evaluation.
- Implement data analysis pipelines that produce statistical quality reports for research and development and support teams.
- Support team efforts to produce meaningful and actionable analytics on company NLP and NLU products.
- Work with teams around the world on products supporting English, German, Spanish, Italian, and other languages.
- Participate in a business-hours-restricted on-call rotation to resolve team tool errors.
- Comply with corporate policies, procedures, and security standards while performing assigned duties.
Benefits
- Medical, dental, and vision benefits.
- Health Savings Accounts.
- Health Care and Dependent Care Flexible Spending Accounts.
- Disability benefits.
- Life insurance.
- Voluntary benefits.
- Paid absences.
- Retirement benefits.
- Relocation assistance may be authorized.
- Company-coordinated and paid travel expenses for required initial on-site onboarding, subject to the company travel policy.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
