Senior Software Development Engineer in Test

Qualia•Austin, TX

22h•Remote

About The Position

Qualia is hiring a senior software engineer focused on quality infrastructure for intelligent, AI-driven products. The role builds production-grade test infrastructure, agentic validation systems, AI/ML evaluation harnesses, observability platforms, and engineering tooling. It helps evolve quality architecture from primarily deterministic automation toward a combination of traditional automation and agentic validation, where AI agents can understand application state, execute workflows, use engineering tools, validate outcomes, investigate failures, and provide evidence for conclusions. The position involves solving loosely defined technical problems and making decisions about frameworks, architecture, evaluation strategy, observability, and validation across the testing pyramid. The engineer works closely with Engineering and Quality leadership, understands the systems under test in depth, and mentors engineers on reliable testing, tooling, infrastructure, and AI-assisted validation. The role is based in Austin, Texas, with remote candidates welcome. Qualia is a remote-first development organization and also has offices in San Francisco, California, and Concord, New Hampshire.

Requirements

At least 5 years of experience as an SDET or software engineer building test infrastructure, developer tooling, platform engineering, observability, or release engineering systems; experience should not be primarily writing test cases or automation scripts.
Strong programming ability in JavaScript/TypeScript, including designing backend services, libraries, frameworks, and production-quality infrastructure from scratch.
Strong software engineering fundamentals, including system design, distributed systems, APIs, asynchronous systems, and production operations.
Experience building or scaling CI/CD systems, release orchestration, test infrastructure, or developer platforms.
Experience designing observability/telemetry systems using structured logging, metrics, and distributed tracing.
Hands-on experience with LLMs, AI-assisted engineering, or agentic systems beyond simply using an AI coding assistant, including tool/function calling, agent workflows, and multi-step AI workflows.
Experience or familiarity with LLM and agent evaluation, including evaluation datasets, deterministic and model-based evaluators, and challenges in testing probabilistic/non-deterministic systems.
Strong understanding of the testing pyramid and experience determining where validation belongs across unit, integration, API, UI, performance, or agentic layers.
Comfort mentoring engineers and raising the technical quality bar across teams.

Responsibilities

Build and maintain production-grade test infrastructure, AI/ML evaluation systems, agentic validation harnesses, release orchestration tooling, observability, and performance testing platforms.
Design agentic test harnesses that let AI agents interact with applications, browsers, APIs, databases, logs, telemetry, and CI/CD systems to execute and validate complex workflows based on application state.
Build AI/ML evaluation frameworks for non-deterministic systems, including deterministic assertions, semantic and model-based evaluation, regression datasets, human-in-the-loop review, and measurable quality signals.
Build observability and telemetry into automation and agentic infrastructure using structured logs, metrics, distributed tracing, and execution timelines.
Develop systems that distinguish product, test, infrastructure, and AI/model reasoning failures, correlating failures with logs, traces, deployments, and commits to accelerate root-cause analysis.
Design and build test automation frameworks, including migration from Cypress to Playwright, and own core infrastructure such as test environment provisioning, parallelization, and CI/CD integration at scale.
Apply AI to engineering-quality problems such as test generation, flaky-test triage, failure clustering, and release-risk analysis, with guardrails requiring agent conclusions to be backed by observable evidence.
Partner with Software Engineers, Quality Engineers, SRE, and Platform teams to make capabilities reusable platforms.
Mentor engineers on test architecture, observability, and AI-assisted engineering.