Staff Software Engineer – Testing & Evals Frameworks

ID.me•McLean, VA

1d•$174,000 - $253,000•Onsite

About The Position

This Staff Engineer role sits at the intersection of engineering, testing, AI evaluations, and developer experience. The engineer will own the architecture and strategy for evaluation frameworks, automation tools, and testing infrastructure that enable ID.me engineering teams to ship quickly and safely across functional, end-to-end, LLM/evaluation, and security-focused testing, as well as the CI/CD quality gates used across the organization. The role calls for deep engineering rigor and the judgment to set testing strategy for complex, asynchronous, event-driven systems. The engineer will build widely adopted tooling and standards, raise quality across the engineering organization, mentor senior and mid-level engineers, champion AI-assisted testing, and partner with engineering, product, and security teams to embed quality throughout development of a platform that verifies the identity of millions of users.

Requirements

10 or more years of experience as a software engineer in modern web applications and distributed systems, or equivalent experience: 10 or more years with a Master's degree, or a PhD with 5 or more years.
Proficiency in Python, TypeScript, JavaScript, or Java.
Hands-on experience with Playwright, Jest, API automation, or evaluation frameworks for software or AI systems.
Experience integrating automated tests into CI/CD pipelines, such as GitHub Actions or ArgoCD.
Proven ability to lead cross-team technical initiatives and influence engineering standards.
Track record of mentoring senior engineers and raising quality standards across teams.
Strong written and verbal communication skills across engineering, product, and leadership.

Nice To Haves

Experience with REST APIs, microservices, distributed and event-driven architectures, and workflow orchestration such as Temporal.
Knowledge of OAuth 2.0, OIDC, SAML 2.0, SCIM 2.0, enterprise identity providers such as Okta, Entra ID, Auth0, or Ping, RBAC/ReBAC systems such as Zanzibar or SpiceDB, and multi-tenant authorization.
Experience with authentication and authorization testing, secrets management, tenant isolation, least-privilege access, and identity verification, KYC, PII, or compliance validation.
Experience with canary or phased rollouts, automated rollback, contract testing such as Pact, and performance or load testing tools such as Locust, k6, or Gatling.
Experience with Java/Spring Boot or Ruby on Rails, GCP, Docker, Kubernetes, observability, or chaos engineering.
Experience using or building agentic tools for test generation, code review, and test planning.
Experience building evaluation frameworks for LLM agents, including correctness graders, LLM-as-judge, human-in-the-loop evaluations, and benchmark dataset curation.
Familiarity with agentic frameworks such as Claude API/Anthropic SDK, BrainTrust, LangChain, LangGraph, CrewAI, or similar.
Experience with production monitoring for AI systems, including behavioral drift detection, output sampling, or shadow scoring.
Red-teaming or adversarial testing experience for AI models or agents.
Experience with RAG pipelines, including data chunking and pre-retrieval data quality gates.
Background in identity verification, fraud detection, or regulated industries.
Experience with observability tools such as Datadog or OpenTelemetry.

Responsibilities

Set the architecture and long-term strategy for automation frameworks and testing infrastructure, including Playwright, Python, Jest, and API automation, and ensure comprehensive, reliable coverage across the organization.
Define end-to-end and integration testing strategies for complex, asynchronous, event-driven, and eventually consistent systems, including workflow orchestration platforms such as Temporal.
Build and promote testing-in-production practices, including canary deployments, progressive or phased rollouts, synthetic monitoring, and automated rollback triggers.
Set standards for automated testing of authentication and authorization flows, OAuth 2.0, OIDC, RBAC/ReBAC, secrets management, and tenant isolation.
Integrate automated testing into CI/CD pipelines, such as GitHub Actions, across the organization to provide fast, reliable feedback.
Build evaluation pipelines for AI-assisted features and LLM agents, and apply AI tools to accelerate test authoring, test plan generation, and code review.
Partner with Security, Platform, and Product teams on testing strategy, testable code structure, coverage gaps, and quality checks throughout the development lifecycle.
Mentor senior and mid-level engineers in test architecture, testing strategy, and quality practices.
Debug and identify root causes across frontend, backend, APIs, and distributed services, and drive fixes to completion.
Define and track reliability, quality, and evaluation accuracy metrics, such as time to detect, time to remediate, and output quality scores.

Benefits

Comprehensive medical insurance
Dental insurance
Vision insurance
Health savings account
Flexible spending accounts, including medical, limited-purpose, dependent-care, and commuter accounts
Basic and voluntary life and AD&D insurance
401(k) with company match
Parental leave
Unlimited paid time off, subject to the PTO policy
Eight company-wide holidays
Short- and long-term disability insurance
Accident and critical illness insurance
Referral bonus policy
Employee assistance program
Pet insurance
Travel assistant program
Wellbeing and childcare discounts
Benefit advocates
Learning and development benefit