Senior Software Engineer, Observability
About The Position
As a Senior Software Engineer on Roku’s Observability team, you will help design, develop, and scale the platform used across Roku engineering teams to alert and monitor thousands of services. You will build and evolve Roku’s metrics stack in Go and Rust on top of open-source technologies, and work with tools including Prometheus, Grafana, Thanos, Loki, Tempo, and OpenTelemetry to improve performance, reliability, and usability. The platform processes terabytes of data daily and manages hundreds of millions of active series. You will use AI tools and agents in your daily engineering and operational work, and help improve how the team engineers with AI. You will collaborate with infrastructure and platform teams, automate deployment and configuration, and contribute improvements to open-source projects. The role also includes leading design discussions, mentoring engineers and interns, and helping shape the platform’s technical direction. The Observability team is part of Roku’s Cloud Technology Infrastructure organization. It builds a metrics stack using Go and Rust and open-source technologies, and works with the Cloud Platform team toward unified, cloud-agnostic infrastructure. The team works with foundational infrastructure such as Kubernetes, Istio, and Envoy, as well as CNCF-supported observability projects. Roku describes its working approach as hybrid: teams generally work in the office Monday through Thursday, with Fridays generally flexible for remote work, except where a role or assigned office requires five-day attendance.
Requirements
- Significant experience, typically around 8 years, in Infrastructure Engineering and/or DevOps, with a focus on observability platforms.
- Hands-on experience using AI tools and agents in day-to-day development and operations work.
- Practical experience building with Large Language Models, such as tool/function calling, agents, MCP, RAG, and evaluating LLM output.
- Programming experience, typically around 4 years, in Go and/or Rust.
- Proven experience designing and operating large-scale observability platforms processing metrics, logs, and traces.
- Expertise deploying, operating, and extending open-source observability tools at scale, such as Prometheus, Grafana, Thanos, Loki, Tempo, Cortex, Mimir, or the ELK stack.
- Familiarity with open standards such as OpenTelemetry and OpenMetrics.
- Deep understanding of Kubernetes and Istio architectures and ability to contribute improvements for scalability and integration.
- Hands-on experience with AWS, GCP, and/or Azure.
- Strong ability to engage with customers and stakeholders to deliver tailored observability solutions.
- B.S. or M.S. degree in Computer Science, Engineering, or equivalent experience.
Nice To Haves
- Experience with performance-sensitive systems.
- Strength in either Go or Rust and willingness to grow in the other.
Responsibilities
- Design and build high-performance metrics-stack components in Go and Rust to handle terabytes of data daily and hundreds of millions of active series.
- Use AI tools and agents in development and operations, including code writing, review and testing, operational automation, troubleshooting, and incident response; help shape the team’s AI practices.
- Enhance and optimize Roku’s observability stack and introduce Roku-specific features, including AI-assisted capabilities for platform users.
- Work with the Platform team to identify, standardize, and improve observability tools as part of a cohesive solution.
- Improve the stability, performance, and reliability of the observability platform.
- Build automation around observability tools to simplify developer onboarding for dashboards, alerts, traces, and related capabilities.
- Contribute tools and improvements to the CNCF open-source ecosystem.
- Partner with engineering teams to deliver tailored solutions, improve out-of-the-box experience, and drive innovation.
- Lead design discussions, mentor engineers and interns, and help shape the platform’s technical direction.
Benefits
- Health insurance, including medical, dental, and vision options, subject to location and role availability.
- Equity awards.
- Life insurance.
- Disability benefits.
- Parental leave.
- Wellness benefits, including global mental health and financial wellness support and resources.
- Paid time off in accordance with local leave policies and personal needs.
- Commuter benefits, where available.
- Retirement options such as 401(k) or pension, depending on location.
- Accident benefits, where available.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
