Core Speech AI Evaluation Engineer

Solventum • Pittsburgh, PA

1d

About The Position

The Core Speech AI Evaluation Engineer will establish quality benchmarks and processes for AI-produced medical notes, including automated LLM-as-judge evaluation. The role implements data analysis pipelines that generate statistical quality reports for research and development and support teams, and contributes actionable analytics on the company’s NLP and NLU products. The engineer will collaborate with international teams on products supporting multiple languages, including English, German, Spanish, and Italian, and participate in a business-hours-restricted on-call rotation to resolve team tool errors.

The position supports Solventum’s Health Information Systems Business. The work location is remote or hybrid, with travel of up to 5% possible. Relocation assistance may be authorized. New hires whose start date is October 1, 2025 or later must travel to a designated company location for onsite onboarding; the company coordinates and pays related travel expenses under its travel policy. Applicants must be legally authorized to work in the country of employment without employment visa sponsorship.

Requirements

• Bachelor’s degree or higher and 4 years of data analytics experience, or high school diploma/GED and 8 years of data analytics experience.
• Experience in AI-model quality analysis.
• Legally authorized to work in the country of employment without sponsorship for employment visa status.

Nice To Haves

• Experience in linguistics and/or human-computer interaction.
• Experience with natural language processing/understanding.
• Experience with medical/clinical software development.
• Experience in evaluation design, including rater training, blinding, inter-rater agreement, sample sizing, and identifying noise.
• Specific experience in AI-model quality and safety analysis.
• Skill in working across team boundaries toward a shared goal.

Responsibilities

• Lead efforts to establish quality benchmarks and processes for AI-produced medical notes, including automated LLM-as-judge evaluation.
• Implement data analysis pipelines to produce statistical quality reports for research and development and support teams.
• Support team efforts to produce meaningful and actionable analytics on company NLP and NLU products.
• Work with teams around the world on products supporting languages including English, German, Spanish, and Italian.
• Participate in a business-hours-restricted on-call rotation to resolve team tool errors.
• Comply with corporate policies, procedures, and security standards while performing assigned duties.

Benefits

• Medical, dental, and vision benefits.
• Health Savings Accounts.
• Health Care and Dependent Care Flexible Spending Accounts.
• Disability benefits.
• Life insurance.
• Voluntary benefits.
• Paid absences.
• Retirement benefits.
• Company-coordinated and paid travel expenses for required onsite onboarding, in accordance with the company travel policy.
• Relocation assistance may be authorized.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level