Staff Platform Engineer

Blackhawk Network•Pleasanton, CA

22h•Hybrid

About The Position

Blackhawk Network is seeking an exceptional Staff Platform Engineer specializing in Observability to join our platform engineering team. This role combines deep technical expertise in modern observability tools and practices with strong enablement and advocacy skills. You will be the cornerstone of our observability platform, building world-class monitoring and observability solutions while serving as a trusted advisor and mentor to engineering teams across the organization. As a Staff Platform Engineer focused on Observability, you will architect, configure, and maintain our enterprise observability stack while empowering teams to leverage these tools effectively. You will act as a developer advocate for the o11y domain, bridging the gap between complex observability technologies and engineering teams through hands-on enablement, training, and collaboration. Your work will be essential in ensuring our payment systems maintain the highest standards of reliability, performance, and operational excellence.

Requirements

Bachelor's degree in Computer Science, Computer Information Systems, or related field (or equivalent experience)
6+ years of hands-on experience in Platform Engineering, DevOps, Site Reliability Engineering, or similar roles
Expert-level knowledge of modern observability tools including New Relic, Splunk, Coralogix, Datadog, or similar enterprise o11y platforms
Deep understanding of observability pipelines and data processing frameworks (e.g., OpenTelemetry, Fluentd, Vector, Logstash)
Proven experience configuring and optimizing modern observability stacks in production environments at scale
Strong expertise in AWS cloud platforms and services, particularly CloudWatch, X-Ray, AWS Config, and CloudTrail
Demonstrated ability to enable and mentor teams through training, documentation, and hands-on guidance
Experience with infrastructure as code tools (Terraform, CloudFormation) for managing observability infrastructure
Solid understanding of distributed systems, microservices architectures, and containerization (Docker, Kubernetes)
Proficiency in scripting languages such as Python, Bash, or similar for automation and tooling
Deep knowledge of observability pillars: metrics, logs, traces, and events
Experience instrumenting applications using APM agents, OpenTelemetry SDKs, and custom metrics
Understanding of query languages (PromQL, SPL, NRQL, LogQL) for data analysis and alerting
Expertise in building effective dashboards and visualization strategies
Knowledge of service mesh technologies (Istio, Linkerd) and their observability capabilities
Familiarity with CI/CD pipelines and integrating observability into deployment processes
Experience with Java-based applications and application frameworks (Apache, Tomcat)
Understanding of networking fundamentals, cloud security, and compliance requirements
Excellent communication skills with the ability to explain complex technical concepts to diverse audiences
Strong mentoring and coaching abilities, with a passion for enabling others to succeed
Proven track record of building collaborative relationships across engineering teams
Self-motivated with strong problem-solving skills and analytical thinking
Ability to thrive in a fast-paced, agile environment with changing priorities
Commitment to documentation, knowledge sharing, and building sustainable solutions
Experience presenting to technical and non-technical stakeholders

Nice To Haves

AWS Certifications such as Solutions Architect (Associate/Professional), DevOps Engineer Professional, or SysOps Administrator
Experience with chaos engineering and resilience testing practices
Knowledge of SLI/SLO/SLA frameworks and error budget management
Experience with FinOps practices and cost optimization for observability tools
Contributions to open-source observability projects or active participation in o11y communities
Experience with event-driven architectures and serverless technologies (Lambda, Step Functions, SNS/SQS)
Background in building internal developer platforms or platform-as-a-product initiatives
Familiarity with GitOps practices and tools (ArgoCD, Flux)
Experience with synthetic monitoring and user experience monitoring (RUM)
Knowledge of machine learning applications in observability (AIOps, anomaly detection)

Responsibilities

Serve as the primary observability advocate, educating and enabling engineering teams on best practices for instrumentation, monitoring, and troubleshooting
Design and deliver training programs, workshops, and hands-on sessions to upskill teams on modern observability practices
Partner with development teams to implement effective instrumentation strategies and improve system observability
Create comprehensive documentation, runbooks, and self-service guides that empower teams to leverage the observability platform independently
Act as a trusted technical advisor, working closely with teams to diagnose complex production issues and optimize application performance
Build and nurture a community of practice around observability, fostering knowledge sharing and continuous improvement
Define and evolve observability standards, patterns, and best practices across the organization
Evaluate and integrate emerging observability tools and technologies to continuously improve our monitoring capabilities
Collaborate with SRE, DevOps, and development teams to ensure observability is embedded into the software development lifecycle
Drive initiatives to improve mean time to detection (MTTD) and mean time to resolution (MTTR) through better observability practices
Mentor platform engineers and contribute to the growth of observability expertise within the organization