Agentic AI Test Engineer

Comcast • Mount Laurel, NJ

19h • Onsite

About The Position

The AI Agentic Team designs, builds, and runs intelligent agents end to end. We own agent logic, workflows, integrations, production deployment, and ongoing performance. By combining build and operations, we deliver agentic systems that are reliable, secure, and scalable in real-world use. We are building the next generation of our chat experience, transitioning from a human-assisted workflow to a fully agentic, AI-driven system. As an AI Agentic Software Engineer, you will create re-usable AI agents that will be used across Xfinity applications. Your initial focus will be to enable agentic capabilities at our Xfinity Assistant (chat application) with a strong passion for customer experience and a commitment to building intuitive, high-quality interactions.

Requirements

• 5+ years of experience as an SDET or Automation Engineer.
• Strong technical background in building and maintaining custom automation frameworks.
• Ability to work independently on complex coding tasks while collaborating effectively in an Agile environment.
• Expert-level hands-on coding in Python.
• Solid understanding of Agent Evaluation techniques and using LLMs as a judge.
• Proven experience in Web and API automation.

Nice To Haves

• TypeScript
• Java
• React
• Streamlit
• CI/CD pipeline management (Jenkins/Concourse)

Responsibilities

• Design and implement automated evaluation frameworks that utilize Large Language Models to assess the quality, accuracy, and safety of Agent responses.
• Perform hands-on validation of AI Agents within complex environments, moving beyond static assertions to intelligent, semantic validation.
• Automate the testing of real-time data streams and LLM outputs to ensure consistency and prevent regression in Agent behavior.
• Develop and maintain automation across TypeScript, Java, and React environments to support various application layers.
• Build high-performance automated tests for modern web interfaces and RESTful APIs, ensuring end-to-end system reliability.
• Help build or maintain internal quality dashboards and tools using frameworks like Streamlit.
• Integrate automated tests into CI/CD pipelines (e.g., Jenkins, Concourse) to ensure continuous quality feedback throughout the development lifecycle.
• Actively participate in technical code reviews, focusing on script efficiency, maintainability, and clean code principles.
• Use logs and observability tools to perform deep-dive root cause analysis on automation failures and system defects.

Benefits

• Best-in-class Benefits
• Options, expert guidance and always-on tools, that are personalized to meet the needs of your reality – to help support you physically, financially and emotionally through the big milestones and in your everyday life.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level