Senior Software Engineer - AI Automation

Roku•San Jose, CA

20h•Hybrid

About The Position

Roku is seeking a hands-on, systems-oriented Senior Software Engineer in Test (Sr. SDET) to join their Browse and Discovery Team. This role will own quality for APIs and data pipelines, focusing on automated test frameworks, containerized quality checks, CI/CD workflows, and infrastructure as code. The engineer will utilize agentic AI as a primary mode of working, driving agents to write and maintain integration tests, keep suites healthy, triage failures, and debug production issues. This involves building the tools, harnesses, and evaluations that ensure these AI-driven workflows are trustworthy. The position is ideal for someone who treats agent design as an engineering discipline, grounding agents in context, integrating them with systems, proving their reliability through automation and evaluation, and creating reusable components for other teams.

Requirements

5+ years of experience in software testing and automation, software engineering, or adjacent domains, with strong software engineering fundamentals and the ability to build production-grade systems.
Expertise in Python or Java with familiarity in the other; experience with C/C++ or another systems language is a plus.
Experience designing test plans, test cases, and automated test frameworks for APIs, services, and data pipelines.
Hands-on experience with LLM-based systems, including prompt design, retrieval, tool use, memory handling, and agent orchestration patterns.
Experience building and maintaining retrieval and context pipelines, agent frameworks, and MCP servers or equivalent function-calling architectures.
Experience with CI/CD tooling such as GitLab runners, GitHub Actions, Jenkins, or Travis CI.
Experience with cloud platforms — AWS (preferred), GCP, or Azure — REST APIs, and containerization and orchestration tools such as Docker and Kubernetes.
Experience with infrastructure as code: Terraform or CloudFormation.
Working knowledge of Linux and Bash scripting.
Experience with observability, evaluation, experimentation, and feedback loops for AI systems in production, including monitoring tools such as DataDog, Prometheus, or Grafana.
Ability to work independently, manage ambiguity, move quickly, and deliver incrementally in a fast-paced environment.
Strong analytical and problem-solving skills, excellent communication and collaboration skills, and sound engineering judgment.
Bachelor's or master's degree in Computer Science, Computer Engineering, Electrical Engineering, Data Science, or a related technical field — or equivalent engineering experience.

Nice To Haves

Experience with C/C++ or another systems language is a plus.
AWS (preferred)

Responsibilities

Design and develop automated test frameworks for APIs and data pipelines, and containerize automated quality checks for complex orchestrated services.
Use agentic AI workflows day to day — authoring and maintaining integration tests, keeping test suites healthy, triaging failures, and debugging production issues.
Build the context and retrieval pipelines that ground these agents in the right code, schemas, telemetry, and business logic, and keep them aligned as those sources evolve.
Implement tool-calling and MCP-style integrations so agents can safely act on the systems around them — test runners, CI, ticketing, logs, and data services.
Build and maintain CI/CD workflows and infrastructure as code using Terraform or CloudFormation, so deployments and test execution are repeatable and auditable.
Establish evaluation, observability, and monitoring for the signals that matter here: test reliability and flake rate, mean time to triage, agent task success rate, latency, and cost.
Build safeguards that improve production readiness and reliability — controlled rollouts, drift detection, and mechanisms that prevent error amplification in multi-step agent workflows.
Create reusable templates, modular components, and paved-path patterns that accelerate adoption across teams, and partner with development, data engineering, and product teams to improve end-to-end testing and release processes.