Senior Software Engineer, Observability

Roku • New York, NY

2h • $268,950 - $403,750 • Hybrid

About The Position

As a Senior Software Engineer on Roku's Observability team, you will design, develop, and scale the platform Roku engineering teams use to monitor and alert on thousands of services. You will build and evolve Roku's metrics stack in Go and Rust on top of open-source technologies, working with tools such as Prometheus, Grafana, Thanos, Loki, Tempo, and OpenTelemetry. The platform processes terabytes of data daily and manages hundreds of millions of active series, so the work emphasizes performance, reliability, and usability.

AI is central to the team's engineering approach. You will use AI tools and agents in daily development and operations, including code writing, review, testing, operational automation, troubleshooting, and incident response. You will also help improve the team's AI workflows and develop AI-assisted capabilities for platform users.

You will collaborate with infrastructure and platform teams to standardize and automate observability tooling, support a unified cloud-agnostic infrastructure, and improve the experience for engineers using the platform. The role also includes contributing to open-source projects, leading design discussions, mentoring engineers and interns, and helping shape the platform's technical direction.

Roku generally has teams work in the office Monday through Thursday, with Fridays generally flexible for remote work. Some roles or office locations may require five days a week in the office. The position is eligible for health insurance, equity awards, life insurance, disability benefits, parental leave, wellness benefits, and paid time off. Annual salary ranges vary by location: California, $251,000–$403,750; Illinois, $220,900–$325,200; and New York, $268,950–$403,750.

Requirements

• Significant infrastructure engineering and/or DevOps experience, typically around 8 years, focused on observability platforms.
• Hands-on experience using AI tools and agents in day-to-day development and operations work.
• Practical experience building with large language models, such as tool or function calling, agents, MCP, RAG, and evaluating LLM output.
• Solid programming experience, typically around 4 years, in Go and/or Rust.
• Proven experience designing and operating large-scale observability platforms processing metrics, logs, and traces.
• Experience deploying, operating, and extending open-source observability tools at scale, such as Prometheus, Grafana, Thanos, Loki, Tempo, Cortex, Mimir, or the ELK stack.
• Familiarity with open standards such as OpenTelemetry and OpenMetrics.
• Deep understanding of Kubernetes and Istio architectures and the ability to contribute improvements for scalability and integration.
• Hands-on experience with AWS, GCP, and/or Azure.
• Ability to engage with customers and stakeholders to deliver tailored observability solutions.
• B.S. or M.S. in Computer Science, Engineering, or equivalent experience.

Nice To Haves

• Experience with performance-sensitive systems.
• Strong experience in either Go or Rust and interest in growing in the other.

Responsibilities

• Design and build high-performance metrics-stack components in Go and Rust to handle terabytes of data daily and hundreds of millions of active series.
• Use AI tools and agents in development and operations, including writing, reviewing, and testing code; automating operational tasks; and accelerating troubleshooting and incident response.
• Help shape and improve the team's AI engineering practices.
• Enhance and optimize Roku's observability stack, including Roku-specific features and AI-assisted capabilities for platform users.
• Work with the Platform team to identify, standardize, and improve observability tools.
• Improve the observability platform's stability, performance, and reliability.
• Build automation that simplifies developer onboarding and use of dashboards, alerts, traces, and other observability tools.
• Contribute open-source tools and improvements to the CNCF ecosystem.
• Partner with engineering teams to deliver tailored solutions and improve the out-of-the-box experience.
• Lead design discussions, mentor engineers and interns, and help shape the platform's technical direction.

Benefits

• Health insurance, including healthcare that may include medical, dental, and vision coverage, depending on location and role.
• Equity awards.
• Life insurance.
• Disability benefits.
• Parental leave.
• Wellness benefits, including access to mental health and financial wellness support and resources.
• Paid time off, subject to local leave policies and personal needs.
• Local benefits may include accident coverage, commuter benefits, and retirement options such as a 401(k) or pension.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior