Senior Software Engineer, Observability
About The Position
As a Senior Software Engineer on Roku’s Observability team, you’ll help design, develop, and scale the platform used across Roku engineering teams to alert and monitor thousands of services. You’ll build and evolve Roku’s metrics stack in Go and Rust on top of open-source technologies, and work hands-on with Prometheus, Grafana, Thanos, Loki, Tempo, and OpenTelemetry to optimize performance, reliability, and usability. The platform ingests terabytes of data daily and manages hundreds of millions of active series. AI is central to the role. You’ll use AI tools and agents in daily development and operations, including building, testing, reviewing, and operating systems, automating operational work, and accelerating troubleshooting and incident response. You’ll also help improve the team’s AI engineering workflows and develop AI-assisted capabilities for platform users. You’ll collaborate with infrastructure and platform teams to improve tooling, standardize solutions, automate deployment and configuration, and make observability easier for developers to use. The role includes contributing to open-source projects, partnering with engineering teams, leading design discussions, mentoring engineers and interns, and helping shape the platform’s technical direction. Roku’s hybrid approach generally has teams working in the office Monday through Thursday, with Fridays generally flexible for remote work, except where a role or assigned office requires five days of attendance.
Requirements
- Typically around 8 years of experience in Infrastructure Engineering and/or DevOps, focused on observability platforms.
- Hands-on experience using AI tools and agents in day-to-day development and operations work.
- Practical experience building with large language models, such as tool or function calling, agents, MCP, RAG, and evaluating LLM output.
- Typically around 4 years of programming experience in Go and/or Rust; strength in one language is welcome, with willingness to grow in the other.
- Proven experience designing and operating large-scale observability platforms for metrics, logs, and traces.
- Expertise deploying, operating, and extending open-source observability tools at scale, such as Prometheus, Grafana, Thanos, Loki, Tempo, Cortex, Mimir, or the ELK stack.
- Familiarity with open standards such as OpenTelemetry and OpenMetrics.
- Deep understanding of Kubernetes and Istio architectures, with the ability to contribute improvements for scalability and integration.
- Hands-on experience with AWS, GCP, and/or Azure.
- Strong ability to engage with customers and stakeholders to deliver tailored observability solutions.
- B.S. or M.S. degree in Computer Science, Engineering, or equivalent experience.
Nice To Haves
- Experience with performance-sensitive systems.
Responsibilities
- Design and build high-performance metrics-stack components in Go and Rust to process large volumes of data.
- Use AI tools and agents in development and operations, and help shape the team’s AI engineering practices.
- Enhance and optimize Roku’s observability stack and introduce Roku-specific features, including AI-assisted platform capabilities.
- Work with the Platform team to identify, standardize, and improve observability tools.
- Improve the stability, performance, and reliability of the observability platform.
- Build automation that simplifies developer onboarding and the use of dashboards, alerts, and traces.
- Contribute tools and improvements to the CNCF open-source ecosystem.
- Partner with engineering teams to deliver better out-of-the-box observability experiences and drive innovation.
- Lead design discussions, mentor engineers and interns, and help shape the platform’s technical direction.
Benefits
- Health insurance, including healthcare options that may include medical, dental, and vision coverage
- Equity awards
- Life insurance
- Disability benefits
- Parental leave
- Wellness benefits
- Paid time off
- Global mental health and financial wellness support and resources
- Commuter benefits
- Retirement options, including 401(k) or pension, depending on location
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
