Software Development Engineer in Test (SDET) – ML & LLM Systems

Comcast • Washington, DC

2d • Hybrid

About The Position

This team operates at the forefront of applied machine learning, focusing on niche and highly specialized technologies such as NLP pipelines, large language model (LLM) evaluations, and ML quality frameworks. Unlike teams that primarily train or consume models, this group specializes in evaluating, validating, and measuring LLM behavior, playing a critical role in ensuring model reliability, performance, and responsible deployment. Engineers on the team gain deep, market‑relevant expertise by continuously adapting to rapidly evolving LLM architectures, evaluation metrics, and benchmarks. The work is fast‑moving, highly technical, and offers exposure to cutting‑edge challenges that few teams encounter, making it an exciting environment for engineers who want to stay ahead of the curve in the LLM and AI evaluation space.

Requirements

• 2+ years of industry experience as a Software Development Engineer in Test (SDET).
• Strong experience with API testing and solid understanding of RESTful APIs.
• Hands-on experience with test processes, test design, test case creation, and defect management systems.
• Strong coding skills with hands-on experience in Java and Spring Boot, and Python.
• Prior Experience with testing Machine Learning models, NLP services, Large Language Models (LLMs) and LLM evaluation frameworks.
• Demonstrated ability to design, build, and maintain test automation frameworks.
• Comfortable working in Linux environments; hands-on experience with container technologies such as Docker and Kubernetes.
• Experience building or working with CI/CD pipelines using tools such as Jenkins.
• Working knowledge of AWS services, including S3.
• Experience with NoSQL databases such as MongoDB.
• Familiarity with Agile development methodologies and working within cross-functional Scrum teams.
• Strong written and verbal communication skills, with the ability to collaborate effectively across teams.
• Highly detail-oriented, analytical, and proactive, with a strong passion for quality, automation, and continuous improvement.
• Bachelor's Degree

Nice To Haves

• Prior Comcast experience preferred

Responsibilities

• Partner with product owners, engineers, and internal stakeholders to understand product functionality, technical requirements, and quality expectations; analyze testing needs and define appropriate automation strategies.
• Design, develop, and maintain robust automated test code using software engineering best practices, ensuring reusability, reliability, and scalability.
• Develop and enhance automated test suites for functional, regression, and performance testing across machine learning models, NLP systems, and supporting backend services.
• Execute automated and manual tests as part of build validation to ensure expected behavior and maintain release quality.
• Design and implement robust test strategies and automation frameworks to support evolving services, workflows, and ML model testing requirements.
• Develop and maintain automated validation pipelines for data ingestion, data quality checks, schema validation, and anomaly detection across large-scale datasets.
• Perform model validation and verification, including accuracy benchmarking, regression testing, drift detection, and reproducibility checks before and after deployment.
• Investigate failing tests, diagnose root causes, and implement fixes to improve test stability and confidence.
• Review automation code contributed by peers, providing constructive feedback aligned with internal coding and quality standards.
• Coordinate QA and production build deployments in collaboration with engineering and infrastructure teams.
• Support triage and resolution of integration and production issues to ensure system reliability and rapid recovery.
• Communicate clearly and effectively with team members and leadership regarding risks, test results, priorities, and quality improvements.
• Collaborate with engineering, product, and leadership partners to continuously improve test coverage, automation effectiveness, and end‑to‑end customer experience.
• Stay current with emerging test tools, frameworks, and industry trends, and help evolve internal testing practices.
• Contribute to additional team initiatives and responsibilities as business needs evolve.
• Regular, consistent and punctual attendance.
• Must be able to work nights and weekends, variable schedule(s) and overtime as necessary.

Benefits

• Medical & Dental
• 401(k) Savings Plan
• Generous paid time off
• Life Milestones - from adoption assistance, childcare resources, pet insurance, and more, Comcast supports you at all life stages.
• Courtesy Services - We offer all of our full-time employees in serviceable areas free digital TV and internet.
• Discount tickets for Universal Resorts, including theme park tickets and onsite hotel rooms.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level