Lead Software Development Engineer in Test

Alteryx • Irvine, CA

4d • $165,000 - $185,000

About The Position

The Lead Software Development Engineer in Test will work on test infrastructure that powers agent evaluation and observability for Ask Alteryx. The role partners with engineering, product, and data science to measure and improve agent quality. The position includes developing comprehensive test strategies and cases, designing and implementing tests across the test pyramid, building scalable automation frameworks, conducting performance and load testing, and creating test reports. The Lead SDET also reports regressions, provides feedback on design and testability, and collaborates as a member of an autonomous, cross-functional agile team.

Requirements

• Experience evaluating AI agents, including multi-step task completion, tool selection, tool-call accuracy, and failure recovery.
• Working knowledge of agent frameworks; LangChain is preferred, with ability to understand and debug context handling, tool execution, and agent state.
• Hands-on experience with LangSmith or similar platforms for evaluation datasets, custom evaluators, experiments, and comparisons across agent changes.
• Experience with agent observability, using traces and telemetry to investigate quality issues, latency, token usage, and cost.
• Strong Python software engineering skills to build and extend evaluation tooling, metrics pipelines, and reporting systems.
• Applied statistics and data analysis skills to assess variability, compare experiments, and distinguish meaningful improvements from noise.
• Experience developing evaluation datasets and scoring methods that combine deterministic checks, human feedback, and LLM-based judges.
• Ability to contribute at a technical lead level, guiding evaluation practices, mentoring engineers, integrating evaluations into CI/CD, and translating production failures into regression coverage.

Nice To Haves

• JavaScript or TypeScript experience.
• Modeling experience.

Responsibilities

• Analyze requirements to produce comprehensive test strategies and detailed test cases.
• Design and implement manual and automated tests across unit, component, integration, and end-to-end levels.
• Design, build, and enhance scalable automated test frameworks for feature testing and AI-driven product workflows.
• Develop and maintain UI and API test automation using tools such as Selenium, pytest, and Playwright.
• Conduct performance and load testing to ensure product scalability.
• Build test reports using reporting tools such as Allure.
• Report regressions to the corresponding development team.
• Influence design decisions by providing timely feedback on design documents and testability.
• Collaborate with engineers, product management, UX, and other agile team members.
• Evaluate AI agents, including multi-step task completion, tool selection, tool-call accuracy, and failure recovery.
• Use agent observability traces and telemetry to investigate quality issues, latency, token usage, and cost.
• Build and extend evaluation tooling, metrics pipelines, and reporting systems.
• Develop evaluation datasets and scoring methods using deterministic checks, human feedback, and LLM-based judges.
• Guide evaluation practices, mentor engineers, integrate evaluations into CI/CD, and turn production failures into regression coverage.

Benefits

• Medical, dental, and vision coverage.
• 401(k) with company match.
• Paid parental leave and caregiver leave.
• Flexible time off.
• Mental health support and wellness reimbursement.
• Career development and education assistance.
• Monthly Connectivity Plus stipend of $150 for remote work-related expenses.
• Annual $200 home office reimbursement.
• Target annual bonus of 15% of base salary, based on individual and company performance.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed