Principal Manager - AI Quality Engineering

Tokio Marine Group of Companies•Lower Merion Township, PA

About The Position

In support of the VP of Quality Assurance, this role works closely with IT and business leaders to formulate and implement comprehensive QA Automation strategies and policies. The Principal Manager, Quality Assurance provides strategic leadership and hands-on technical expertise in Automation and Performance engineering. This pivotal role is responsible for building and advancing intelligent, Java/Python-based automation frameworks and performance testing solutions while driving AI-augmented testing practices. The role combines strong leadership, strategic planning, technical oversight, and active hands-on contribution to drive innovation in quality engineering practices across the organization.

Requirements

12+ years of progressive Dev/QA experience with a proven track record of leadership and management of teams for at least 5 years.
5+ years of hands-on experience building and maintaining automation frameworks in Python and/or Java.
Strong proficiency in Selenium WebDriver, pytest/TestNG, and at least one API testing tool (RestAssured, Postman, ReadyAPI).
Experience with CI/CD tools (Azure DevOps, Jenkins, GitHub Actions) and version control (Git).
Demonstrated experience leading large-scale automation initiatives with measurable coverage and ROI outcomes.
Working knowledge of AI/ML concepts including supervised/unsupervised learning, model evaluation metrics, data quality, and bias detection.
Experience using AI coding assistants (GitHub Copilot, Claude Code, or equivalent) for development productivity.
Experience with data analysis libraries (Pandas, NumPy) and SQL for test data management and validation.
Excellent communication skills with ability to present technical strategies to executive leadership.

Nice To Haves

Bachelor’s Degree in Computer Science, Engineering, Information Technology, or related field.
3+ years of experience working in P&C insurance or financial services.
Experience with AI/ML testing frameworks and tools (Great Expectations, Evidently AI, Fairlearn, SHAP).
Experience with LLM/GenAI applications (prompt engineering, LangChain, RAG architectures).
Experience with cloud platforms (Azure, AWS, GCP) and containerization (Docker).
Experience with performance testing tools (JMeter, Gatling, LoadRunner).
Experience migrating legacy automation (UFT/QTP) to modern frameworks.
Familiarity with MLOps practices (MLflow, model monitoring, data drift detection).
Understanding of P&C insurance business processes (policy lifecycle, billing, claims, underwriting).
AWS Certified Machine Learning — Specialty
Microsoft Certified: Azure AI Engineer Associate
TensorFlow Developer Certificate (Similar hands-on coding certification)
(Any) Industry recognized AI certification

Responsibilities

Heads the Automation Engineering and Performance Engineering functions for the QA organization, providing strategic direction, technical leadership, and hands-on guidance to build scalable, efficient, and intelligent automation frameworks.
Owns and drives the Enterprise business process automation coverage target across critical applications, establishing baseline metrics, gap analysis, and phased delivery roadmaps.
Architects and maintains end-to-end process automation that spans multiple integrated systems (policy administration, claims, billing, document management, data reporting) validating complete business transactions.
Drives innovation in test automation by actively evaluating, piloting, and implementing AI/ML-powered testing capabilities including intelligent test case generation, self-healing test scripts, AI-assisted defect prediction, and automated root cause analysis.
Evaluates and integrates AI coding assistants (GitHub Copilot, Claude Code, Cursor) into QA development workflows to accelerate script development, code review, framework and process modernization.
Designs and implements AI/ML model testing frameworks covering data quality validation, model performance testing (accuracy, precision), bias and fairness testing, and production drift monitoring — critical for insurance industry AI applications in insurance domain applications.
Leverages Large Language Models (LLMs) and Generative AI for test case generation from requirements, synthetic test data creation, automated test documentation, and intelligent test prioritization.
Implements Retrieval-Augmented Generation (RAG) solutions for building QA knowledge bases and test asset repositories.
Serves as a hands-on technical contributor in Python/Java-based automation development.
Leverages strong proficiency in Python along with key automation packages including Selenium WebDriver, Playwright, pytest, RestAssured, Appium, Pandas, NumPy, and AI/ML libraries (scikit-learn, TensorFlow/PyTorch, LangChain, SHAP, Fairlearn) to develop scalable automation solutions.
Leads the migration of legacy automation scripts (UFT/QTP) to modern Automation frameworks supporting ReactJS, Unqork and modern web application architectures.
Designs and implements Page Object Model (POM) frameworks, data-driven testing architectures, and API testing solutions (RestAssured, Postman) for comprehensive multi-layer validation (UI, API, database, integration).
Establishes CI/CD pipeline integration for automated test execution using Azure DevOps, Jenkins, or GitHub Actions with parallel execution capabilities.
Implements test data management strategies including synthetic data generation, data masking, and environment provisioning.
Formulates and implements QA strategies and policies to ensure product quality and compliance with industry standards and regulations applicable to P&C insurance.
Supervises the development and execution of test plans, including manual and automated testing, to identify defects and ensure product quality.
Works closely with Application Development, Engineering, Data Operations, and Customer Service to ensure quality standards are integrated throughout the product lifecycle
Identifies potential risks related to product quality, develops mitigation strategies, and ensures these are communicated and managed effectively.
Drives continuous improvement initiatives within the QA department, leveraging industry best practices, emerging technologies, and data-driven methodologies to enhance quality and efficiency.
Establishes AI/ML governance frameworks for model validation, explainability, and regulatory compliance in collaboration with Risk, Compliance, and Actuarial teams.
Ensures conversion of business requirements into verifiable use cases/test cases and maintenance tasks.
Provides guidance for the planning and execution of User Acceptance Testing (UAT).
Ensures that products meet all relevant regulatory and industry standards.
Leads and mentors a team of QA engineers, fostering a culture of continuous learning, innovation, and technical excellence.
Designs and executes team upskilling programs for AI/ML testing, modern automation frameworks, and cloud-native testing practices.
Builds high-performance teams through effective training, mentoring, delegating, and providing constructive feedback.
Manages vendor relationships for QA tools, AI testing platforms, and contractor augmentation.
Complies with proper internal controls as necessary to conduct job functions and/or carry out responsibilities and/or administrative activities.
Performs special duties and other projects as may be assigned.
Establishes and builds strong working relationships and partnerships with IT domain units and senior management.
Leads change initiatives and operations.
Sets Service Level Agreements, ensures SLAs are met; when SLAs are not met, leads improvement efforts.
Serves as focal point of communication for major service outages/escalations.