Director, AI Automation Engineering

Impiricus•Atlanta, GA

4d•$185,000 - $210,000•Remote

About The Position

The Director, AI Automation Engineer is a player-coach role responsible for owning the quality and reliability layer for all AI Organization shipments. This involves hands-on building of harnesses, evaluation frameworks, and release gates to ensure the quality and reliability of AI agents and tools at scale. The role covers both internal AI platform work (Sage and its tools) and customer-facing AI across Ascend and DocUpdate. This is the first dedicated hire in this function, with the expectation of growing into a leadership role managing a small automation team. The primary goal is to enable AI Engineers and builders to ship efficiently without placing a QA burden on the core build team. Daily partnerships will be with AI Engineers, AI Solutions Architects, and product teams to establish quality metrics, automate validation, and create usable standards for both technical and non-technical users building within Sage.

Requirements

5+ years of experience building, evaluating, or governing production AI/ML systems, with direct experience in agentic AI, LLM-based automation, or autonomous system safety.
Direct experience evaluating LLM outputs, including RAG eval, prompt regression, and safety/accuracy benchmarking.
Experience building CI/CD pipelines with integrated test gates (e.g., GitHub Actions).
Comfort working across both AI platform infrastructure and product-facing AI features.
Ability to operate independently as a senior individual contributor and communicate quality standards effectively to both engineers and non-engineers.
Proficiency in Python and standard testing frameworks (e.g., pytest).

Nice To Haves

Experience with agent testing, bot lifecycle management, or AI agent orchestration platforms.
Familiarity with LLM observability and evaluation tooling (e.g., LangSmith, Braintrust, custom harnesses).
Background in healthcare, pharma, or regulated environments where quality gates are critical.
Prior experience establishing a QA or automation function from the ground up, with an interest in growing into a team leadership role.

Responsibilities

Own the infrastructure that continuously verifies whether autonomous agents are taking the intended actions, both before and after production.
Stage, test, and lifecycle-manage bots and agent workflows prior to production release.
Establish baseline quality metrics and coverage standards for AI initiatives.
Build the infrastructure required to run automated research chains and agent evaluations at scale.
Create automated evaluation frameworks for LLM outputs, focusing on accuracy, safety, bias, and regression.
Implement RAG evaluation, prompt regression testing, and golden-set benchmarking.
Define and enforce quality thresholds that AI Engineers and builders must meet before shipping.
Partner with AI Researchers and Engineers on the design of evaluations for new model and agent patterns.
Build the release-gating infrastructure that governs when an agent is trusted for production deployment, including staged rollouts and rollback capabilities.
Develop AI automation solutions to ensure agentic coding delivers highly functioning software while maintaining quality across the product portfolio.
Integrate testing into GitHub Actions (or equivalent) for AI model and agent releases.
Maintain existing test suites and ensure they remain current with evolving models, prompts, and tools.
Collaborate with AI Solutions Architects on evaluation criteria for workflow automations built in the field.
Partner with Principal AI Product on launch readiness and QA for AI projects across the organization.
Support product teams (Ascend, DocUpdate, Platform) with AI feature validation without owning their roadmaps.
Document standards, runbooks, and patterns to enable self-service basic testing for non-engineers building in Sage.