Senior Software Development Engineer in Test (SDET), QA - AI Systems

Nexxa.AI•Sunnyvale, CA

About The Position

Nexxa is seeking a Senior / Staff SDET to own quality for its AI agent systems. The role combines test automation with hands-on manual and exploratory testing of agents that plan, call tools, and take multi-step actions in industrial environments. The engineer will write production-grade code to build evaluation harnesses, simulation environments, and CI/CD quality gates, while also probing agent behavior, reviewing traces, and validating releases when automation cannot determine whether an agent acted appropriately. The SDET will work closely with ML engineers, backend engineers, and Forward Deployed Engineers to define quality standards for agents operating in high-stakes industrial settings, and to build and run systems and processes that measure quality continuously. The role includes establishing evaluation datasets, metrics, quality bars, and release processes, as well as mentoring engineers and advocating for testability, observability, and tracing in agent architecture.

Requirements

At least 5 years as an SDET, QA engineer, or software engineer with a strong testing focus, including ownership of test strategy and automation for complex distributed systems.
Demonstrated experience in both manual and automated testing, including exploratory testing, test planning, and writing high-quality bug reports.
Strong programming skills, preferably in Python, and a software engineering mindset focused on clean, maintainable, well-reviewed code for frameworks, data pipelines, and tooling.
Experience building test frameworks and infrastructure from the ground up, not only writing test cases within an existing framework.
Hands-on experience with CI/CD systems such as GitHub Actions, GitLab CI, or Jenkins, and containerized test environments such as Docker or Kubernetes.
Experience testing APIs and microservices, including contract, integration, and end-to-end testing.
Hands-on experience testing LLM-based products, chatbots, or AI agents, and understanding of why traditional deterministic assertions can break down for generative systems.
Practical experience with evaluation frameworks or tooling such as promptfoo, DeepEval, RAGAS, or LangSmith, or a track record of building evaluation tools.
Understanding of LLM agents, including prompting, tool/function calling, context management, RAG, memory, and orchestration frameworks.
Experience designing test data and labeled datasets, including sourcing, sampling, and managing dataset drift over time.
Familiarity with LLM-specific failure modes, including hallucination, prompt injection, context poisoning, tool misuse, goal drift, and non-determinism.
Comfort operating in ambiguity and defining correctness for tasks without a single right answer.
Strong written communication skills for turning ambiguous quality signals into clear, actionable findings for engineering and product stakeholders.

Nice To Haves

Experience with human-in-the-loop evaluation workflows, including labeling pipelines, inter-rater reliability, and rubric design.
Experience with performance and load testing tools such as k6 or Locust, and chaos or fault-injection testing.
Background in ML or data science sufficient to read model evaluations and assess statistical significance.
Experience red-teaming or performing adversarial/security testing on ML systems.
Familiarity with LLM application observability and tracing tools such as LangSmith, Arize, Langfuse, or Weights & Biases, and general observability stacks such as OpenTelemetry, Grafana, or Datadog.
Experience testing systems in industrial, IoT, or operational technology environments, including industrial protocols such as OPC UA, MQTT, or Modbus, or simulation and hardware-in-the-loop setups.
Experience supporting UAT, pilots, or customer-facing validation of complex products.
Experience establishing test and evaluation infrastructure from scratch at a startup or fast-moving team.

Responsibilities

Design, build, and maintain test frameworks, evaluation harnesses, and regression suites for LLM-based agents, covering reasoning quality, tool-call correctness, task completion, and multi-turn coherence.
Build automated pipelines that run evaluations and test suites on model, prompt, and tool-integration changes, and integrate them into CI/CD as enforceable quality gates.
Develop simulators, mocks, and sandbox environments for industrial systems and tool APIs so agents can be tested safely, repeatably, and at scale.
Implement API, contract, integration, and end-to-end tests for agent-dependent services, tools, and orchestration layers.
Develop performance, load, and reliability tests for agent workflows, including failure injection and resilience testing.
Perform hands-on exploratory testing of agent behavior, including new capabilities, ambiguous requests, long multi-turn sessions, and unusual operational scenarios.
Manually review agent traces and outputs for reasoning quality, action appropriateness, and safety, and use findings to improve automated checks.
Execute manual regression and release verification in staging and customer-like environments before new capabilities ship.
Conduct manual red-teaming sessions for prompt injection, jailbreaks, tool misuse, and unsafe actions with security teams, and convert successful attacks into automated test cases.
Support user acceptance testing and pilot rollouts with Forward Deployed Engineers, capturing real-world failures and feeding them back into test suites.
Write test plans, test charters, and bug reports, and maintain manual test documentation where automation is not practical or cost-effective.
Create and manage golden datasets and labeled test sets, including industrial-context edge cases, ambiguous inputs, and adversarial prompts, with tooling for versioning and drift detection.
Design rubrics and run human-in-the-loop evaluations where automated scoring is not sufficiently reliable.
Define and instrument quality metrics such as groundedness, hallucination rate, task success rate, latency/cost tradeoffs, and safety violations, and build dashboards to track them.
Test agent behavior across the full action loop, including planning, tool selection and execution, error recovery, and final output.
Investigate and triage failures involving models, prompts, tools or APIs, and orchestration logic, and turn them into reproducible, actionable bug reports.
Establish quality bars, release criteria, and sign-off processes for agent capabilities, combining automated gates with manual verification.
Decide what to automate and what to keep manual, and continually move repeatable manual work into automation.
Mentor engineers on testing strategies and test-driven practices for probabilistic, LLM-driven systems.
Advocate for testability, observability, and tracing in agent architecture from day one.