Staff Software Engineer, Backend (Observability)

Affirm

7h • $204,000 - $290,000 • Remote

About The Position

Affirm’s Observability team builds the platform engineers use to understand system behavior and performance. It ingests more than 200 TB per day of metrics, logs, and traces from hundreds of services across more than 80 Kubernetes clusters. The team is moving toward wide events, with one context-rich event per request, and is introducing log sampling and observability practices for agentic development and code review.

The Staff Software Engineer will serve as the technical lead for Observability, working primarily hands-on while defining the roadmap, leading cross-functional projects, mentoring six engineers, and partnering with infrastructure and product teams. The role owns telemetry pipeline architecture and technical strategy, while helping ensure reliability, query performance, cost efficiency, operational readiness, and high-quality instrumentation.

Requirements

• At least 8 years of experience designing, developing, and launching backend systems at scale using languages such as Python or Kotlin.
• Extensive experience developing highly available distributed systems using technologies such as AWS, MySQL, Flink, and Kubernetes.
• Experience building or operating high-volume telemetry systems involving metrics, logs, and/or traces, using technologies such as OpenTelemetry, Kafka, and Terraform.
• A well-informed perspective on modern observability practices, including structured and wide events, sampling, cardinality management, and SLOs.
• Experience driving adoption of a platform standard across multiple engineering teams through tooling and engagement.
• Experience delivering major features or system components, or deprecating existing functionality, through technical and execution planning.
• Ability to write high-quality code that is easy for others to understand and use.
• Comfort working through ambiguity, from low-level language idioms to the architecture of large systems.
• A track record of gathering and iterating on feedback from engineering and cross-functional peers.
• Strong verbal and written communication skills for collaboration with a global engineering team.
• Equivalent practical experience or a bachelor’s degree in a related field.

Responsibilities

• Own the architecture of Affirm’s telemetry pipeline, balancing reliability, query performance, and cost at more than 200 TB per day.
• Design log sampling strategies that reduce volume and cost while preserving useful incident-response signals.
• Shape how observability is integrated into agentic development so AI coding agents and code review can identify instrumentation gaps before code ships.
• Set the team’s technical strategy on a year-long horizon and connect it to critical, business-impacting projects.
• Collaborate with product management, design, and analytics throughout the product development lifecycle to manage technical sustainability, risks, and trade-offs.
• Advocate for technical solutions and operational processes that increase the effectiveness of the team.
• Own team operations and availability, including monitoring, triage rotations, playbooks, policies, testing, and alerting for ongoing operations and on-call efforts.
• Set code review and design standards, promote quality and ownership within and beyond the team through writing and tech talks.
• Develop team members through feedback, guidance, and leading by example.
• Mentor a team of six engineers and partner closely with infrastructure and product teams.

Benefits

• 100% subsidized medical coverage for employees and dependents.
• Dental and vision coverage for employees and dependents.
• Monthly stipends for technology spending and health and wellness options.
• Flexible time off and generous holiday calendars.
• Employee stock purchase plan allowing employees to buy Affirm stock at a discount.
• Equity rewards may be included in total compensation.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior