Senior Observability Engineer

Royal Bank of Canada • Edina, MN

1d • $90,000 - $140,000 • Onsite

About The Position

This role involves designing and implementing observability solutions using industry-leading platforms to establish logging standards for comprehensive system visibility. The engineer will create and maintain monitoring dashboards, evaluate and recommend observability tools, and analyze logs, metrics, and traces to proactively identify system issues and performance bottlenecks. A key aspect of the role is developing predictive monitoring strategies using AI tools for anomaly detection and trend analysis, and conducting "what if" analysis to model potential scenarios. The position requires collaboration with cross-functional teams and leadership in observability initiatives, including mentoring junior team members.

Requirements

• 5+ years of experience with observability tools (ELK Stack, Dynatrace, Prometheus, Grafana, Open Telemetry, Jaeger, Aternity and Moog or similar)
• 3+ years of software engineering or infrastructure experience
• Python, Java, GO
• Query languages
• Expert-level knowledge of logging requirements and best practices for enhanced observability
• Demonstrated experience building and optimizing monitoring dashboards
• Proven ability to use observability data to proactively identify and resolve system issues
• Experience using AI tools for anomaly detection and trend analysis
• Alerting
• Linux, Unix

Nice To Haves

• Expertise with Tableau or advanced visualization tools
• Experience in financial technology and financial services environments
• Previous experience using recent AI tools such as Anthropic, OpenAI, Devin, Copilot, etc.
• Experience with OpenShift, Kubernetes and containerized applications
• Background in DevOps or Site Reliability Engineering (SRE)
• Experience conducting "what if" analysis and scenario modeling
• Identify networking slowness and availability issues

Responsibilities

• Design and implement observability solutions using industry-leading platforms, establishing logging standards that enable comprehensive system visibility across all systems
• Create and maintain monitoring dashboards that provide actionable insights into system health and performance, partnering with platform and application teams to integrate observability into architecture
• Evaluate and recommend observability tools and vendors to ensure the organization has access to best-in-class solutions
• Analyze logs, metrics, and traces to proactively identify system issues, performance bottlenecks, and translate observability data into insights about user patterns and system behavior
• Develop predictive monitoring strategies using AI tools to detect anomalies, identify emerging trends, and prevent incidents before they occur
• Conduct "what if" analysis using AI capabilities to model potential scenarios and their impact on system performance
• Apply machine learning-based anomaly detection to identify issues proactively and use predictive analytics to forecast system behavior and prevent failures
• Collaborate with cross-functional teams (Engineering, DevOps, Security, production support, product) to communicate complex observability concepts to both technical and non-technical stakeholders
• Lead observability initiatives and mentor junior team members on best practices while facilitating design and problem-solving discussions across the organization

Benefits

• A comprehensive Total Rewards Program including bonuses and flexible benefits
• competitive compensation
• stock where applicable
• Leaders who support your development through coaching and mentoring opportunities
• Ability to make a difference and lasting impact on system reliability and user experience
• Work in a dynamic, collaborative, progressive, and high-performing team
• RBC’s discretionary variable compensation program

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed