Staff Software Engineer – Testing & Evals Frameworks

ID.me•Mountain View, VA

21h•$174,000 - $229,000•Onsite

About The Position

This Staff Engineer role sits at the intersection of engineering, testing, AI evaluations, and developer experience. The engineer will own the architecture and strategy for evaluation frameworks, automation tools, and testing infrastructure that help ID.me teams ship quickly and safely across functional, end-to-end, LLM/evaluation, and security-focused testing, as well as the CI/CD quality gates used across the organization. The role calls for deep engineering rigor, judgment in setting testing strategy for complex asynchronous and event-driven systems, and the ability to build tooling and standards adopted widely by other teams. The engineer will raise quality across the engineering organization, mentor senior and mid-level engineers, champion AI-assisted testing practices, and partner with engineering, product, and security teams to embed quality throughout development for a platform that verifies the identity of millions of users. ID.me's culture is full-time and in-office; unless a posting explicitly states otherwise, roles are onsite five days per week at an office in McLean, Virginia; Mountain View, California; New York City, New York; or Tampa, Florida.

Requirements

At least 10 years of experience as a software engineer working with modern web applications and distributed systems, or equivalent experience (10+ years with a master's degree, or a Ph.D. with 5+ years).
Proficiency in Python, TypeScript, JavaScript, or Java.
Hands-on experience with Playwright, Jest, API automation, or evaluation frameworks for software or AI systems.
Experience integrating automated tests into CI/CD pipelines, such as GitHub Actions or ArgoCD.
Demonstrated ability to lead cross-team technical initiatives and influence engineering standards.
Track record of mentoring senior engineers and raising quality across teams.
Strong written and verbal communication skills across engineering, product, and leadership.

Nice To Haves

Experience testing REST APIs, microservices, distributed and event-driven architectures, or workflow orchestration platforms such as Temporal.
Experience with OAuth 2.0, OIDC, SAML 2.0, SCIM 2.0, enterprise identity providers (Okta, Entra ID, Auth0, or Ping), RBAC/ReBAC (Zanzibar or SpiceDB), and multi-tenant authorization.
Experience testing authentication and authorization, secrets management, tenant isolation, least-privilege access, identity verification, KYC, or PII/compliance validation.
Experience with canary or phased rollouts, automated rollback, contract testing (Pact), and performance or load testing tools such as Locust, k6, or Gatling.
Experience with Java/Spring Boot or Ruby on Rails; GCP, Docker, or Kubernetes; observability; or chaos engineering.
Experience using or building agentic tools for test generation, code review, and test planning.
Experience building evaluation frameworks for LLM agents, including correctness graders, LLM-as-judge, human-in-the-loop evaluations, or benchmark dataset curation.
Familiarity with agentic frameworks such as the Claude API/Anthropic SDK, BrainTrust, LangChain, LangGraph, CrewAI, or similar.
Experience with production monitoring for AI systems, including behavioral drift detection, output sampling, or shadow scoring.
Experience with red-teaming or adversarial testing of AI models or agents.
Experience with RAG pipelines, including data chunking and pre-retrieval data quality gates.
Background in identity verification, fraud detection, or regulated industries.
Experience with observability tools such as Datadog or OpenTelemetry.

Responsibilities

Own the architecture and long-term strategy for automation frameworks and testing infrastructure, including Playwright, Python, Jest, and API automation, and ensure comprehensive, reliable coverage across the organization.
Define end-to-end and integration testing strategy for complex asynchronous, event-driven, and eventually consistent systems, including workflow orchestration platforms such as Temporal.
Build and champion testing in production through canary deployments, progressive or phased rollouts, synthetic monitoring, and automated rollback triggers.
Set standards for security-focused test automation covering authentication and authorization flows, OAuth 2.0, OIDC, RBAC/ReBAC, secrets management, and tenant isolation.
Integrate automated testing into CI/CD pipelines, such as GitHub Actions, across the organization to provide fast and reliable feedback.
Build evaluation pipelines for AI-assisted features and LLM agents, and apply AI tools to accelerate test authoring, test plan generation, and code review.
Partner with Security, Platform, and Product teams on testing strategy, testable code structure, coverage gaps, and quality checks throughout the development lifecycle.
Mentor senior and mid-level engineers on test architecture, testing strategy, and quality practices.
Debug and find root causes of issues across frontend, backend, APIs, and distributed services, and drive fixes to closure.
Define and track reliability, quality, and evaluation-accuracy metrics, such as time to detect, time to remediate, and output quality scores.

Benefits

Comprehensive medical insurance.
Dental insurance.
Vision insurance.
Health savings account.
Flexible spending accounts for medical, limited-purpose, dependent-care, and commuter benefits.
Basic and voluntary life and AD&D insurance.
401(k) with company match.
Parental leave.
Unlimited paid time off, subject to the PTO policy, including eight company-wide holidays.
Short- and long-term disability insurance.
Accident and critical illness insurance.
Referral bonus policy.
Employee assistance program.
Pet insurance.
Travel assistant program.
Wellbeing and childcare discounts.
Benefit advocates.
Learning and development benefit.