Senior Software Engineer, Observability

Roku • Chicago, IL

1d • Hybrid

About The Position

The Senior Software Engineer will help design, develop, and scale Roku’s observability platform, which engineering teams use to monitor and alert on thousands of services. The role involves building and evolving a metrics stack in Go and Rust on top of open-source technologies, and optimizing tools such as Prometheus, Grafana, Thanos, Loki, Tempo, and OpenTelemetry for performance, reliability, and usability. The platform processes terabytes of data daily and supports hundreds of millions of active series.

AI is part of the team’s engineering and operations workflow. The engineer will use AI tools and agents to develop, review, and test code, automate operational work, and accelerate troubleshooting and incident response, while helping improve the team’s AI practices. The role also includes developing AI-assisted capabilities for platform users, automating deployment and configuration, collaborating with infrastructure and platform teams, and contributing improvements to open-source projects.

The Observability team is part of Roku’s Cloud Technology Infrastructure organization. It works with the Cloud Platform team on a unified, cloud-agnostic infrastructure and builds foundational infrastructure and observability capabilities using open-source and CNCF-supported projects. The team emphasizes ownership, accountability, collaboration, open communication, and rapid iteration. This role includes technical leadership through design discussions, mentoring engineers and interns, and helping shape the platform’s technical direction.

Roku generally follows a hybrid working model: teams typically work in the office Monday through Thursday, with Fridays generally flexible for remote work. Some employees may be required to attend the office five days a week based on their role or assigned location.

Requirements

• Significant experience, typically around 8 years, in Infrastructure Engineering and/or DevOps, focused on observability platforms.
• Hands-on experience using AI tools and agents in day-to-day development and operations work.
• Practical experience building with Large Language Models, such as tool or function calling, agents, MCP, RAG, and evaluating LLM output.
• Solid programming experience, typically around 4 years, in Go and/or Rust.
• Proven experience designing and operating observability platforms that process large volumes of metrics, logs, and traces.
• Experience deploying, operating, and extending open-source observability tools at scale, such as Prometheus, Grafana, Thanos, Loki, Tempo, Cortex, Mimir, or the ELK stack.
• Familiarity with open standards such as OpenTelemetry and OpenMetrics.
• Deep understanding of Kubernetes and Istio architectures, with the ability to contribute improvements for scalability and integration.
• Hands-on experience with AWS, GCP, and/or Azure.
• Strong ability to engage with customers and stakeholders to deliver tailored observability solutions.
• B.S. or M.S. degree in Computer Science, Engineering, or equivalent experience.

Nice To Haves

• Experience with performance-sensitive systems.
• Willingness to grow in either Go or Rust if already strong in the other.

Responsibilities

• Design and build high-performance metrics-stack components in Go and Rust for large-scale data processing.
• Use AI tools and agents in development and operations, and help shape the team’s AI engineering practices.
• Enhance and optimize the observability stack and introduce Roku-specific features, including AI-assisted platform capabilities.
• Work with the Platform team to identify, standardize, and improve observability tools.
• Improve the platform’s stability, performance, and reliability.
• Build automation that simplifies developer onboarding and use of dashboards, alerts, and traces.
• Contribute open-source tools and improvements to the CNCF ecosystem.
• Partner with engineering teams to deliver observability solutions and improve the out-of-the-box experience.
• Lead design discussions, mentor engineers and interns, and help shape the platform’s technical direction.
• Collaborate with infrastructure and platform teams, automate deployment and configuration, and contribute improvements to open-source projects.

Benefits

• Health insurance, including medical, dental, and vision options depending on location and role.
• Equity awards.
• Life insurance.
• Disability benefits.
• Parental leave.
• Wellness benefits, including global access to mental health and financial wellness support and resources.
• Paid time off in accordance with local leave policies and personal needs.
• Commuter benefits, where available.
• Retirement options, including 401(k) or pension, depending on location.
• Local statutory and voluntary benefits, depending on location and role.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior