ML QA Analyst

OnCorps, Inc. • Boston, MA

23d • Hybrid

About The Position

This role is central to how our AI systems learn, improve, and perform. As an ML QA Analyst, you’ll work under the Data Science umbrella and own key tasks across model evaluation, training, and ongoing maintenance for the pipelines and work products built by our DS team. You’ll validate agent behavior, strengthen prompts, ensure high-quality labeled data, and keep our models running reliably in production. It’s an ideal fit for someone with a STEM background, strong analytical instincts, and a passion for precision who wants hands-on experience in a cutting-edge AI environment. This role also offers a clear growth path—successful team members often progress into Data Science roles or transition into Product as Business/Product Analysts.

Requirements

• STEM BA/BS
• High attention to detail
• Ability to think critically and analytically
• Ability to follow and understand a complex ruleset
• U.S. based with work authorization
• Comfortable working in fast-paced environments and meeting tight deadlines

Nice To Haves

• Familiarity with LLMs/Agents is a plus
• Knowledge of SQL is a plus
• Knowledge of Python is a plus

Responsibilities

• Labeling: Owning data labeling for PDF / Excel-based ML ingestion engines across the company. This includes both training-based labeling and live-run corrective labeling.
• Agent Evaluation Review: Review evaluation jobs where agents are underperforming, digging into individual logs and traces and providing documentation of possible prompt weaknesses.
• Agent Regression Testing: Interacting with customer facing “chat” agents. Validating the integrity of an agent’s response to both “happy-path” and adversarial prompts pre-deployment & when model updates are in testing.
• Dashboarding: Owning client-specific dashboard deployments. Working with product/CS stakeholders to deploy client-specific visualizations. Testing dashboards for errors / logical gaps.
• Maintenance-Mode Model Upkeep: Initiating and monitoring routine model retraining for models in a steady-state & fixing job failures in production. Working with Data Science to determine appropriate thresholds for escalation.
• AI Table Annotation

Benefits

• Innovative Environment: Work with cutting-edge AI and machine learning technologies in a company that is at the forefront of financial services innovation.
• Growth Opportunities: As a rapidly growing startup, we offer significant opportunities for professional development and career advancement.
• Collaborative Culture: Be part of a collaborative and inclusive team where your ideas are valued, and your contributions make a real impact.
• Work-Life Balance: Enjoy the flexibility of a hybrid working environment, allowing you to balance your professional and personal life effectively.
• Competitive Compensation: We offer a competitive salary and a comprehensive benefits package.

What This Job Offers

• Job Type: Full-time
• Career Level: Entry Level