Manager AI Quality, Security & Assurance

Westinghouse Electric Company, LLC • Cranberry Township, PA

10d • $130,400 - $163,000 • Hybrid

About The Position

As Manager, AI Quality, Security & Assurance, you will lead a team responsible for ensuring that AI applications, agent-based workflows, and Technology Bricks developed by Data & AI are reliable, secure, and compliant with Westinghouse policies and standards. The team’s work supports the suitability of these systems for use in the nuclear industry. You will map testing to Westinghouse policy, define and apply testing frameworks for AI quality and evaluation, and establish product security assurance practices across the Data & AI portfolio. The team also provides an independent readiness recommendation before release.

The role is hybrid/mobile and based in Cranberry, Pennsylvania. It reports to the Director, AI Security.

The assurance framework includes risk-based classification by intended use, assurance plans, minimum evidence requirements, Definition of Done, readiness reviews, Technology Brick integration into solutions, and exception management. It integrates with the common AI SDLC and applies Westinghouse AI policies and software quality requirements without redefining them.

The manager establishes methods and datasets to evaluate GenAI responses and agent behavior, including grounding, robustness, tool use, human oversight, and regression. This includes setting acceptance thresholds and automating evaluations following model, prompt, or data changes. The role defines test strategies and provides test frameworks and practices integrated into CI/CD pipelines for development teams.

The manager coordinates application security assessments and leads risk-based red teaming of GenAI systems and agents with Corporate Cybersecurity. Findings are tracked through remediation or formal acceptance. Before release, the team reviews testing, evaluation, and security evidence and issues a readiness recommendation, escalating unresolved findings and residual risks to the appropriate owner, who retains formal acceptance. The role also documents reports before turnover and integration of Technology Bricks into existing or legacy systems in accordance with WEC policy.

The role serves as the primary Data & AI counterpart to IT Quality Assurance to align shared testing services, environments, tools, and standards. It engages early in planning so assurance effort and acceptance criteria are included in estimates for new developments and commercial projects. It reports portfolio quality and readiness metrics and improves methods, automation, and standards.

The manager leads a small core team in AI evaluation, test automation, and security assurance, scaling the team through hiring, contractors, and specialist third parties as demand grows. People leadership includes goal setting, performance management, recognition, succession planning, and coaching, mentoring, and upskilling team members in AI evaluation and AI security.

Requirements

• Bachelor’s degree in computer science, software engineering, data science, engineering, or a related discipline.
• 8+ years of experience in software quality engineering, test automation, software assurance, or application security.
• At least 2 years of hands-on experience testing or evaluating ML, GenAI, or agent-based systems.
• At least 3 years leading quality, testing, or assurance teams, including hiring and growing teams.
• Knowledge of LLM and RAG evaluation methods and metrics, hallucination and grounding analysis, agent evaluation, evaluation datasets, and LLM-as-a-judge techniques.
• Experience with test strategy, functional, regression, and performance testing, and test automation integrated into CI/CD; examples include Azure DevOps and Python-based frameworks.
• Knowledge of application security, secure SDLC, and AI-specific threats such as prompt injection, data leakage, and insecure tool use.
• Experience with security scanning tools such as AppScan and with red teaming.
• Ability to define proportionate evidence requirements and make and defend independent readiness recommendations.
• Documentation and audit-ready evidence practices, and knowledge of software quality requirements in regulated environments.
• Ability to challenge development teams constructively and collaborate across Cybersecurity, AI Governance, and business owners.

Nice To Haves

• Familiarity with AI risk frameworks such as the NIST AI Risk Management Framework.

Responsibilities

• Lead a team ensuring AI applications, agent-based workflows, and Technology Bricks are reliable, secure, and aligned with Westinghouse policy and standards.
• Define and maintain an assurance framework covering risk-based classification, assurance plans, minimum evidence requirements, Definition of Done, readiness reviews, Technology Brick integration, and exception management.
• Establish methods, datasets, acceptance thresholds, and automation for evaluating GenAI responses and agent behavior, including grounding, robustness, tool use, human oversight, and regression after model, prompt, or data changes.
• Define test strategies for AI applications and Technology Bricks and provide test frameworks and practices integrated into CI/CD pipelines.
• Coordinate application security assessments and lead risk-based red teaming of GenAI systems and agents with Corporate Cybersecurity; track findings until remediated or formally accepted.
• Review testing, evaluation, and security evidence before release and issue independent readiness recommendations; escalate unresolved findings and residual risks to the applicable owner.
• Document reports before turnover and Technology Brick integration into existing or legacy systems according to WEC policy.
• Act as the primary Data & AI counterpart to IT Quality Assurance and align shared testing services, environments, tools, and standards.
• Engage early in planning to include assurance effort and acceptance criteria in estimates for new developments and commercial projects.
• Report portfolio quality and readiness metrics and improve assurance methods, automation, and standards.
• Lead and scale the AI evaluation, test automation, and security assurance team through hiring, contractors, and specialist third parties as demand grows.
• Oversee team goal setting, performance management, recognition, and succession; coach, mentor, and develop team members.

Benefits

• Medical benefits, potentially including medical, dental, vision, prescription coverage, and a Health Savings Account with employer contribution options.
• Wellness programs and an Employee Assistance Program for employees and household members.
• 401(k) with company match contributions.
• Paid vacations and company holidays.
• Opportunities for flexible work arrangements.
• Educational reimbursement and comprehensive career programs.
• Global recognition and service programs.
• Employee referral program.

What This Job Offers

• Job Type: Full-time
• Career Level: Manager