Director, AI Automation Engineering

Impiricus • Atlanta, NY

3d • Remote

About The Position

The Director, AI Automation Engineer is a player-coach role: own the quality and reliability layer for everything the AI Organization ships, and stay hands-on building the harnesses, eval frameworks, and release gates that make that real. That means automated testing infrastructure, evaluation frameworks, and CI/CD guardrails that let us stage, test, and manage AI agents and tools at scale. Coverage spans internal AI platform work (Sage and the tools built on it) and customer-facing AI across Ascend and DocUpdate. This is the first dedicated hire into a function we expect to grow. The Director sets the bar, does the hardest technical work early, and grows into leading a small automation team as coverage expands. The goal is to keep AI Engineers and builders shipping without loading QA burden onto the core build team. Partnership is daily with AI Engineers, AI Solutions Architects, and product teams shipping AI features: establish baseline quality metrics, automate validation, and make standards usable for engineers and non-engineers building in Sage.

Requirements

• 5+ years building, evaluating, or governing production AI/ML systems, with direct experience in agentic AI, LLM-based automation, or autonomous system safety
• Direct experience evaluating LLM outputs: RAG eval, prompt regression, safety and accuracy benchmarking.
• Experience building CI/CD pipelines with integrated test gates (GitHub Actions or similar).
• Comfortable working across AI platform infrastructure and product-facing AI features.
• Operates independently as a senior IC and communicates quality standards to engineers and non-engineers.
• Proficiency in Python and standard testing frameworks (pytest or equivalent).

Nice To Haves

• Experience with agent testing, bot lifecycle management, or AI agent orchestration platforms is a strong plus.
• Familiarity with LLM observability and eval tooling (LangSmith, Braintrust, custom harnesses, etc.) is a strong plus.
• Background in healthcare, pharma, or regulated environments where quality gates matter is a strong plus.
• Prior experience standing up a QA or automation function from scratch, with appetite to grow into leading a team.

Responsibilities

• Own the infrastructure that continuously verifies whether autonomous agents are taking the actions we intend, before and after they reach production.
• Stage, test, and lifecycle-manage bots and agent workflows before they reach production.
• Establish baseline quality metrics and coverage standards for AI initiatives.
• Build the infrastructure to run automated research chains and agent evals at scale.
• Create automated evaluation frameworks for LLM outputs: accuracy, safety, bias, and regression.
• Implement RAG evaluation, prompt regression testing, and golden-set benchmarking.
• Define and enforce quality thresholds that AI Engineers and builders must meet before ship.
• Partner with AI Researchers and Engineers on eval design for new model and agent patterns.
• Build the release-gating infrastructure that governs when an agent is trusted to act in production, including staged rollouts and rollback authority.
• Develop AI automation solutions to ensure that agentic coding delivers highly functioning software while maintaining quality across the product portfolio.
• Integrate testing into GitHub Actions (or equivalent) for AI model and agent releases.
• Maintain existing test suites and keep them current as models, prompts, and tools change.
• Work with AI Solutions Architects on eval criteria for workflow automations they build in the field.
• Partner with Principal AI Product on launch-readiness and QA for AI projects across the org.
• Support product teams (Ascend, DocUpdate, Platform) on AI feature validation without owning their roadmaps.
• Document standards, runbooks, and patterns so non-engineers building in Sage can self-serve basic testing.

Benefits

• Medical, dental, and vision coverage for you and your dependents
• On-demand healthcare concierge
• Pre-tax savings options for healthcare and dependent care
• Monthly employer contributions to HSA (if enrolled in a high-deductible plan)
• 100% paid short- and long-term disability
• Life and AD&D insurance
• Flexible vacation policy
• Paid parental leave to spend time with your newborn, adopted, or foster child (available after 6 months)
• 401(k) with company match

What This Job Offers

• Job Type: Full-time
• Career Level: Director
• Education Level: No Education Listed