Member of Technical Staff, Quality Systems

Comfy•San Francisco, CA

9d•$180,000 - $220,000•Onsite

About The Position

ComfyUI has a large and diverse user base with a vast number of community-built nodes, leading to unique workflow variations. This role focuses on building automated systems to ensure quality, rather than traditional manual testing. The goal is to create infrastructure that enables rapid, daily releases by proving quality through machines. The mandate is to shift from a release cycle measured in weeks to one measured in days, making quality assurance a machine-driven process.

Requirements

Experience building testing or verification infrastructure relied upon by other engineers.
Experience writing production code; this is an engineering role.
Experience with systems where scripted test coverage was insufficient, employing alternative methods like property-based testing, fuzzing, simulation, generative testing, or evaluation harnesses.
Experience shipping a release gate and understanding its impact on blocking launches.
Proficiency in TypeScript or Python.
Comfort with CI systems and build graphs.
A strong perspective on test flakiness, treating it as a defect.
A startup mindset, thriving in fast-moving environments, taking ownership, and solving problems at scale.

Nice To Haves

Experience testing graphics, canvas, or rendering surfaces (e.g., WebGL, shaders, visual diffing at scale).
Experience building evaluation systems for non-deterministic output (e.g., golden datasets, judge calibration, scored regression tracking).
Experience maintaining an open-source project, particularly with a plugin or extension ecosystem across versions.
Experience using ComfyUI or contributing to it.

Responsibilities

Define and build systems to verify the correctness of ComfyUI, including release gates and coverage of real user journeys.
Develop property-based and generative testing strategies to address the combinatorial complexity of user-created workflows with numerous community nodes.
Create perceptual comparison systems to solve the visual correctness problem, addressing issues in canvas rendering, node layout, and link routing that pixel diffs cannot reliably catch.
Build an evaluation harness to manage non-deterministic output caused by variations in seeds, hardware, and drivers, distinguishing regressions from acceptable variance.
Develop custom node CI to continuously verify the compatibility of third-party nodes, which form a significant part of the ComfyUI ecosystem.
Aggressively utilize AI agents for tasks like exploratory testing and test case generation, implementing checks to reject the majority of AI-produced output.
Publish and own metrics that demonstrate improvements in quality to the team.
Contribute to building evaluation systems for AI-assisted code generation to ensure its quality, once product quality is stable.