Senior Observability Engineer

RBC•Minneapolis, MN

9d•$90,000 - $140,000•Onsite

About The Position

This role focuses on designing, implementing, and maintaining observability solutions to ensure comprehensive system visibility, health, and performance. The engineer will leverage industry-leading platforms, establish logging standards, create insightful dashboards, and evaluate new tools. A key aspect involves analyzing logs, metrics, and traces to proactively identify issues, understand system behavior, and develop predictive monitoring strategies using AI for anomaly detection and scenario modeling. Collaboration with cross-functional teams and mentoring junior members are also integral parts of the position.

Requirements

5+ years of experience with observability tools (ELK Stack, Dynatrace, Prometheus, Grafana, Open Telemetry, Jaeger, Aternity and Moog or similar)
3+ years of software engineering or infrastructure experience
Python, Java, GO
Query languages
Expert-level knowledge of logging requirements and best practices for enhanced observability
Demonstrated experience building and optimizing monitoring dashboards
Proven ability to use observability data to proactively identify and resolve system issues
Experience using AI tools for anomaly detection and trend analysis
Alerting
Linux, Unix

Nice To Haves

Expertise with Tableau or advanced visualization tools
Experience in financial technology and financial services environments
Previous experience using recent AI tools such as Anthropic, OpenAI, Devin, Copilot, etc.
Experience with OpenShift, Kubernetes and containerized applications
Background in DevOps or Site Reliability Engineering (SRE)
Experience conducting "what if" analysis and scenario modeling
Identify networking slowness and availability issues

Responsibilities

Design and implement observability solutions using industry-leading platforms, establishing logging standards that enable comprehensive system visibility across all systems
Create and maintain monitoring dashboards that provide actionable insights into system health and performance, partnering with platform and application teams to integrate observability into architecture
Evaluate and recommend observability tools and vendors to ensure the organization has access to best-in-class solutions
Analyze logs, metrics, and traces to proactively identify system issues, performance bottlenecks, and translate observability data into insights about user patterns and system behavior
Develop predictive monitoring strategies using AI tools to detect anomalies, identify emerging trends, and prevent incidents before they occur
Conduct "what if" analysis using AI capabilities to model potential scenarios and their impact on system performance
Apply machine learning-based anomaly detection to identify issues proactively and use predictive analytics to forecast system behavior and prevent failures
Collaborate with cross-functional teams (Engineering, DevOps, Security, production support, product) to communicate complex observability concepts to both technical and non-technical stakeholders
Lead observability initiatives and mentor junior team members on best practices while facilitating design and problem-solving discussions across the organization