Senior Software Engineer, Observability
About The Position
The Senior Software Engineer will help design, develop, and scale Roku's observability platform, which is used across engineering teams to monitor and alert on thousands of services. The role builds and evolves a custom metrics stack using Go and Rust on open-source technologies, and works hands-on with Prometheus, Grafana, Thanos, Loki, Tempo, and OpenTelemetry to improve performance, reliability, and usability. The platform processes terabytes of data daily and manages hundreds of millions of active series. The engineer will use AI tools and agents daily to develop, test, and operate systems, and contribute to improving the team's AI-enabled engineering practices. The role also collaborates with infrastructure and platform teams, automates deployment and configuration, contributes to open-source projects, and helps provide AI-assisted platform capabilities. Technical leadership includes leading design discussions, mentoring engineers and interns, and shaping the platform's technical direction. Roku generally follows a hybrid work approach: teams typically work in the office Monday through Thursday, with Fridays generally flexible for remote work. Some roles or office locations may require five days of in-office attendance.
Requirements
- Significant experience, typically around 8 years, in Infrastructure Engineering and/or DevOps, focused on observability platforms.
- Hands-on experience using AI tools and agents in day-to-day development and operations.
- Practical experience building with Large Language Models, such as tool or function calling, agents, MCP, RAG, and evaluating LLM output.
- Solid programming experience, typically around 4 years, in Go and/or Rust.
- Proven experience designing and operating observability platforms that process large volumes of metrics, logs, and traces.
- Expertise deploying, operating, and extending open-source observability tools at scale, such as Prometheus, Grafana, Thanos, Loki, Tempo, Cortex, Mimir, or the ELK stack.
- Familiarity with open standards such as OpenTelemetry and OpenMetrics.
- Deep understanding of Kubernetes and Istio architectures, and ability to contribute improvements for scalability and integration.
- Hands-on experience with AWS, GCP, and/or Azure.
- Strong ability to engage customers and stakeholders to deliver tailored observability solutions.
- B.S. or M.S. degree in Computer Science, Engineering, or equivalent experience.
Nice To Haves
- Experience with performance-sensitive systems.
- Willingness to grow in Rust or Go when already strong in the other language.
Responsibilities
- Design and build high-performance metrics-stack components in Go and Rust on open-source technologies.
- Use AI tools and agents in daily development and operations, including writing, reviewing, and testing code, automating operational work, and accelerating troubleshooting and incident response.
- Help shape and improve the team's engineering workflows with AI.
- Enhance and optimize Roku's observability stack and introduce Roku-specific features, including AI-assisted capabilities for platform users.
- Work with the Platform team to identify, standardize, and improve observability tools as part of a cohesive solution.
- Improve the stability, performance, and reliability of the observability platform.
- Build automation that simplifies developer onboarding to observability tools, including dashboards, alerts, and traces.
- Contribute open-source tools and improvements to the CNCF ecosystem.
- Partner with engineering teams to deliver solutions that improve out-of-the-box experiences and drive innovation.
- Lead design discussions, mentor engineers and interns, and help shape the platform's technical direction.
Benefits
- Health insurance, including healthcare options such as medical, dental, and vision, subject to location and role availability.
- Equity awards.
- Life insurance.
- Disability benefits.
- Parental leave.
- Wellness benefits, including global mental health and financial wellness support and resources.
- Paid time off and other leave in accordance with local policies.
- Potential commuter and retirement benefits, including 401(k) or pension options, depending on location and availability.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
