AI Automation Engineer (Quality) NC Hybrid

Cisco•Research Triangle Park, NC

2d•$128,600 - $184,900•Hybrid

About The Position

Cisco’s AI and Automation portfolio team builds and operates Circuit, the central AI engine powering productivity and process automation across Cisco. The team works at the intersection of applied research and production, developing technology for performance, reliability, and functional quality for more than 80,000 monthly users. The AI Automation Engineer owns and improves the automated testing ecosystem that helps keep the Circuit platform reliable, scalable, and production-ready across releases. The role transforms regression testing from manual processes to AI-driven, fully automated test suites for new and legacy features, and designs and owns end-to-end, API, and performance automation. It also builds scalable test frameworks, integrates testing into CI/CD pipelines, and uses AI-assisted coding tools while checking generated tests for stability, correctness, and security compliance. This is a hybrid position.

Requirements

Bachelor's degree with at least 4 years, Master's degree with at least 3 years, or PhD with at least 1 year of related experience in Computer Science, AI/ML, or a related technical field.
Experience in test automation and regression framework development, with a deep understanding of functional quality assurance.
Experience with Playwright (TypeScript/JavaScript) for E2E/UI automation and/or Python with Pytest or Robot Framework; familiarity with Selenium.
Experience with REST API testing; familiarity with API contract testing. GraphQL testing is a plus.
Practical and conceptual knowledge of software development lifecycles, CI/CD pipelines such as GitHub Actions or Jenkins, and automated testing best practices.
Experience with AI-assisted coding tools such as GitHub Copilot, Cursor, or Claude Code, along with critical thinking and Agile/Git workflows to ensure high-quality, scalable test automation.

Nice To Haves

Experience testing AI/ML-powered web applications, including LLM features, chatbots, or AI agent workflows.
Hands-on performance/load testing experience with k6 or JMeter and familiarity with AI/LLM performance metrics such as TTFT, inter-token latency, and token throughput.
Experience in AI Ops/MLOps or building and fine-tuning custom LLMs.
Experience with observability and reporting tools such as Splunk, Grafana, or Prometheus.
Familiarity with Docker and Kubernetes for test environment management and with Google Cloud Platform (GCP).
Track record of leading technical projects and mentoring junior engineers.
Strong communication and collaboration skills, including the ability to influence cross-functional stakeholders.
Experience building and productizing agentic applications with a focus on evaluation and business acceptance.

Responsibilities

Architect, build, and maintain automated regression and end-to-end test suites for the Circuit platform using Playwright, Python/Pytest, and Selenium, replacing manual QA with scalable automation.
Design and automate REST and GraphQL API test suites covering functional correctness, contract validation, error handling, and performance, including AI/model-serving endpoints under concurrent load.
Plan and execute load, stress, and soak tests using k6 or JMeter; track response-time percentiles, throughput, error rates, and AI-specific metrics such as Time to First Token and inter-token latency.
Use AI agents and AI-assisted coding tools to generate and maintain regression suites for new and legacy features; assess generated code for stability and ensure it meets security standards before merging.
Develop testing approaches for AI/LLM features, validating model response quality, behavioral consistency, and end-to-end AI agent workflows across reasoning and action layers.
Integrate automated tests into CI/CD pipelines for continuous validation; build dashboards and reports and communicate quality metrics, risk assessments, and defect trends to engineering leadership.
Lead automation initiatives by defining standards, mentoring peers, and collaborating with development, product, and UX teams to embed automation-first testing into the development lifecycle.

Benefits

Medical insurance
Dental insurance
Vision insurance
401(k) plan with Cisco matching contribution
Paid parental leave
Short- and long-term disability coverage
Basic life insurance
Cisco restricted stock unit grants may be available to eligible employees
10 paid holidays per full calendar year
One floating holiday for non-exempt employees
Paid birthday day off
Paid year-end holiday shutdown
Four paid personal wellness days
Paid vacation time for non-exempt employees; exempt employees participate in a flexible vacation program with no defined usage limit, subject to availability and business limitations
80 hours of sick time off on hire and each January 1, with up to 80 hours of unused sick time carried forward
Additional paid time away may be requested for critical or emergency family issues
Optional 10 paid volunteer days per full calendar year
Annual bonuses may be available for non-sales roles