Senior AI Quality Assurance Engineer

Bottomline•Lebanon, PA

About The Position

Are you ready to transform the way businesses pay and get paid? Bottomline is a global leader in business payments and cash management, with over 35 years of experience and moving more than $16 trillion in payments annually. We're looking for passionate individuals to join our team and help drive impactful results for our customers. If you're dedicated to delighting customers and promoting growth and innovation - we want you on our team! Senior AI Quality Assurance Engineer Role Overview An AI Quality Assurance Engineer ensures the accuracy, reliability, and compliance of AI systems and models before deployment. This role is critical for validating AI-driven applications, detecting defects, and safeguarding against risks such as bias, hallucinations, and non-deterministic behavior. The position involves designing robust test strategies for machine learning models and collaborating with data scientists, ML engineers, and product teams to maintain high-quality standards.

Requirements

Bachelor’s in Computer Science, Software Engineering, or related field.
3–7 years in QA, with at least 2 years in AI/ML testing.
Familiarity with non-deterministic testing and AI-specific QA challenges.
Proficiency in Python and testing frameworks (e.g., PyTest, Selenium).
Knowledge of ML frameworks (TensorFlow, PyTorch) and MLOps tools.
Experience with CI/CD pipelines and cloud platforms (AWS, Azure, GCP).
Strong analytical and problem-solving abilities.
Excellent communication and collaboration skills.

Nice To Haves

Understanding of ethical AI principles and bias testing.
Familiarity with low-code/no-code automation tools and AI-driven QA platforms.
Knowledge of regulatory compliance for AI systems.

Responsibilities

Develop and execute comprehensive test plans for AI models and systems.
Perform manual and automated tests to validate AI functionality under various scenarios.
Conduct bias detection, edge-case testing, and stress testing for AI models.
Design and implement automated testing frameworks for AI workflows.
Utilize AI-powered QA tools and self-healing test scripts to improve efficiency.
Ensure AI systems adhere to ethical guidelines, regulatory standards, and internal policies.
Maintain documentation such as Generative AI Bill of Materials (BoM) for transparency and audits.
Work closely with ML engineers, data scientists, and product managers to resolve defects.
Participate in code reviews and contribute to continuous improvement initiatives.
Analyze test results, identify defects, and track quality metrics.
Provide risk assessments and recommendations for AI deployments.