Senior Quality Engineer, Data & AI Platform

M3•Lawrenceville, GA

11h

About The Position

The Senior Quality Engineer - Data & AI sits at the intersection of data engineering, machine learning product delivery, and quality assurance. This role is responsible for validating AI-powered and data-driven features across M3’s hospitality accounting platform, with primary ownership of quality for the Data & AI team’s active products. Unlike traditional QE roles focused on deterministic software, this position requires quality thinking applied to systems that produce probabilistic outputs - where “passing” is defined by accuracy thresholds, not binary correctness. The right candidate understands how AI products fail differently and is motivated to build the testing practice that prevents those failures from reaching customers. This is an individual contributor role with significant cross-functional responsibility, including collaboration with Engineering, Product, Data, and external contractor teams.

Requirements

Minimum of 5 years of experience in software testing or quality engineering, with at least 2 years focused on data-intensive or AI/ML-enabled products.
Bachelor’s Degree in Computer Science, IT, MIS, or equivalent combination of education and experience.
3+ years of hands-on experience with Microsoft SQL Server for data validation and testing; experience using Azure DevOps for test management and release coordination.
2+ years of API testing experience (Postman, Swagger, or SoapUI); experience writing and executing test cases for ETL processes and data warehouse environments.
Hands-on experience with Databricks, Delta Lake, or equivalent Lakehouse platforms and Python for data validation strongly preferred.
Experience validating ML model outputs and familiarity with LLM evaluation, prompt regression testing, or generative AI quality workflows preferred.
Familiarity with dbt, Great Expectations, MLflow, or equivalent pipeline-level assertion and experiment tracking tools preferred.
Working experience with automation frameworks such as Katalon or Playwright; familiarity with USALI and hospitality GL structure a plus.
Understanding of Agile/Scrum methodologies; strong written and verbal communication skills.
CSQA or ISTQB certification strongly preferred; certifications in AI/ML quality or data engineering are welcomed.

Nice To Haves

Hands-on experience with Databricks, Delta Lake, or equivalent Lakehouse platforms and Python for data validation strongly preferred.
Experience validating ML model outputs and familiarity with LLM evaluation, prompt regression testing, or generative AI quality workflows preferred.
Familiarity with dbt, Great Expectations, MLflow, or equivalent pipeline-level assertion and experiment tracking tools preferred.
Familiarity with USALI and hospitality GL structure a plus.
CSQA or ISTQB certification strongly preferred; certifications in AI/ML quality or data engineering are welcomed.

Responsibilities

Design and execute test strategies for AI/ML-enabled features; define acceptance thresholds (precision, recall, F1) in partnership with Engineering and Product.
Validate AI-generated outputs - mappings, anomaly flags, LLM summaries - against domain-grounded acceptance criteria; build regression suites to detect concept drift, label drift, and feature degradation.
Evaluate LLM outputs for accuracy, hallucination, and format compliance; build prompt regression test suites to catch behavior changes when model versions or system prompts are updated.
Write data validation logic in Python or SQL against tables across Bronze, Silver, and Gold layers; design and automate data contract tests covering schema, null rates, referential integrity, and row counts.
Implement automated validation checkpoints in Databricks pipelines; apply statistical testing methods to pipeline output validation.
Define and instrument production monitoring checks for model performance: alert thresholds, confidence degradation, and data drift.
Perform functional, regression, integration, and exploratory testing across releases, enhancements, and defect fixes using manual and automated approaches.
Design, develop, maintain, and execute automated test scripts using frameworks such as Katalon and Playwright; support CI/CD quality gates.
Develop and execute test cases for ETL processes, data warehouse workflows, and API validation.
Review application logs and monitoring tools to identify and track defects; coordinate builds, deployments, and software migrations for planned releases and hotfixes.
Maintain and utilize testing tools including Azure DevOps, SQL Server, Postman/Swagger/SoapUI, and automation frameworks.
Participate in Agile ceremonies; collaborate with Product, Engineering, Data, and external contractor teams to ensure comprehensive coverage and aligned acceptance criteria.
Communicate quality risks early and in writing; develop and maintain a QE runbook for AI/ML products.
Adhere to secure testing practices and compliance requirements associated with the assigned Technical Security Level.
Other duties as assigned.