Automation Test Engineer
About The Position
Skyward is seeking an Automation Test Engineer to support development of an AI/ML platform using generative AI to reduce user burden across the Centers for Medicare and Medicaid Services (CMS). The role focuses on creating reliable, scalable test automation for web applications, APIs, and AI-driven features, and on integrating quality controls throughout AI development and release processes. The engineer will work with AI/ML, engineering, data science, and product teams to validate models and datasets, test functional and operational behavior, monitor bias and drift, and help ensure releases meet reliability and fairness expectations. The position also contributes to responsible AI practices and helps establish automation and AI-testing standards. Skyward describes its culture as people-centered, collaborative, growth-oriented, and guided by integrity, compassion, curiosity, unconventional thinking, and authenticity. Skyward supports flexible working hours and remote opportunities. Applicants must be able to obtain and maintain a Public Trust security clearance, and employment is contingent on acceptable background investigation results.
Requirements
- At least 5 years of experience in software QA or automation engineering, including at least 1 year focused on AI/ML product testing.
- Strong proficiency in at least one automation framework, such as Selenium, Cypress, or Playwright, and a scripting language; Python is preferred.
- Hands-on experience testing AI models, including validation metrics, dataset versioning, and model explainability.
- Familiarity with AI observability tools such as MLflow, TensorBoard, or similar.
- Understanding of model bias, fairness audits, and compliance considerations in government or regulated environments.
- Experience working in Agile DevSecOps environments with modern CI/CD tooling such as Jenkins, GitHub Actions, or GitLab.
- Strong collaboration skills and clear communication across engineering, data science, and product teams.
- Ability to obtain and maintain a Public Trust security clearance.
- Acceptable results from a background investigation.
Nice To Haves
- Previous experience supporting the Centers for Medicare and Medicaid Services (CMS).
Responsibilities
- Design and maintain robust, scalable automated test frameworks for web applications, APIs, and AI-driven features.
- Develop automated pipelines for model validation, bias detection, and drift monitoring in machine-learning systems.
- Define and execute test plans covering functional, regression, performance, and AI-specific testing.
- Partner with AI/ML engineers to create test datasets and simulate real-world user behavior and edge cases.
- Implement CI/CD-integrated quality gates to ensure AI releases meet defined reliability and fairness metrics.
- Mentor junior engineers and help establish best practices in intelligent automation and AI testing.
- Advocate for ethical AI standards and contribute to cross-functional discussions about responsible AI.
Benefits
- Medical, dental, and vision insurance.
- 15 days of paid leave.
- 7 days of sick leave.
- 2 days of bereavement leave.
- 11 paid federal holidays.
- Up to 40 hours for jury duty.
- 401(k) with a 4% employer contribution and no vesting period.
- Up to 4 weeks of paid paternity and maternity leave.
- Company-provided laptop.
- $5,000 per year for professional development.
- $600 per year for technical supplies and equipment.
- $2,000 referral bonus.
- Life and disability insurance.
- HSA and FSA.
- Legal Shield and ID Shield voluntary benefits.
- Flexible working hours and remote opportunities.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Mid Level
Education Level
No Education Listed
