Associate Director AI Product Engineer - Financial Institution Operations & Technology

BRG

7d•$180,000 - $250,000

About The Position

Berkeley Research Group is building a Financial Services Operations & Technology practice focused initially on Commercial and Consumer Banking. The practice combines banking expertise, technology, and AI to help clients improve operations. The hands-on AI Product Engineer will focus on evaluation—determining whether AI products are accurate, reliable, and safe for banking operations. The role builds automated tests, investigates failures, provides evidence for release decisions, and contributes to product development. Working with banking experts and engineers, the engineer defines correct results, measures performance, and turns findings into product improvements. Evaluation is the core responsibility from early development through live use; client stakeholders retain release and risk-acceptance authority. Preferred locations are New York, Chicago, Washington, D.C., Dallas, and Boston. Other locations may be considered case by case. The salary range is $180,000–$250,000. Job title and level will be determined based on qualifications and experience. Candidates must be able to verify their legal right to work in the United States without employer sponsorship now or in the future.

Requirements

At least 5 years of working experience, including at least 1 year of experience in AI evaluation.
Demonstrated ownership of AI evaluations that informed real release decisions, including test design, expert-reviewed answers, scoring, and failure investigation.
Strong Python skills and experience building automated test software, connecting data sources, and recording repeatable results across product versions.
Ability to select representative samples, interpret error rates and uncertainty, and distinguish genuine improvement from weak tests or inconsistent scoring.
Experience building or improving AI products used in operations, with practical controls for sensitive data, output accuracy, and human review.
Experience in financial services or another regulated environment, with the ability to translate business requirements into practical tests and controls.
Strong communication and teamwork skills, ability to mentor colleagues, and willingness to travel to client sites as needed.
Ability to verify legal right to work in the United States without employer sponsorship now or in the future.

Nice To Haves

Banking experience in lending, payments, servicing, collections, or default operations.
Experience independently evaluating AI products, checking automated scoring against expert judgment, or testing security and fairness in financial institutions.
Experience evaluating document review, summaries, information search, or automated workflows, including comparisons across AI services and conventional software.
Consulting or product delivery experience, including client discovery, implementation planning, and user adoption.

Responsibilities

Define what each AI product must get right, how errors could affect customers or operations, and what evidence is needed before use.
Build reusable software to run tests, score results, and compare versions, integrating testing into everyday development.
Work with engineers to reproduce failures, identify causes, verify fixes, and contribute product code where needed to improve accuracy and control.
Set pass criteria and scoring rules with business and risk teams, distinguishing minor defects from errors that must block release.
Maintain representative test cases, including incomplete records and rare, high-impact errors, and reserve unseen examples for final testing.
Coordinate with banking experts to establish expected answers and resolve scoring disagreements; measure factual accuracy, missing information, incorrect additions, and source support.
Check automated scoring against expert review, including reviewer agreement and whether AI-based scoring misses important errors.
Compare product versions using the same tests and report results by task and error severity, including sample sizes and uncertainty.
Test whether systems find the right information and whether answers reflect that information; distinguish source problems from answer errors.
Check extracted facts against original records; independently verify amounts, dates, and totals; and test whether citations support important claims.
Test when the product should answer, request clarification, or refer work to a person, and verify that reviewers can detect and correct material errors.
Test for invented facts, unsupported recommendations, unfair outcomes, sensitive-information disclosure, and attempts to bypass safeguards.
Protect client data, restrict access and permitted actions, use approved services, and verify safeguards in practice.
Document repeatable test results, unresolved failures, and limitations; recommend release, restricted use, or further development; and support independent review and required client approvals.
Sample live outputs, track error rates and user corrections, add confirmed failures to repeatable tests, and trigger investigation or suspension when quality falls below agreed standards.
Explain findings clearly, coach colleagues, develop reusable tests and controls, and support proposals, technical estimates, and recruiting.