AI Operations Engineer

BlytzPay • Draper, UT

6d

About The Position

We are seeking an AI Operations Engineer to join our team. In this role, you will be responsible for building agentic skills using agentic coding, troubleshooting issues when internal users are unable to train the agent, managing human-in-the-loop evaluations, and contributing to architectural strategy regarding memory, context, and tool-triggering.

Requirements

• Deep understanding of LLM memory management, context windows, and tool interaction/orchestration.
• Familiarity with data science principles, including data sanitization, evaluation metrics, and data pipelines.
• Exceptional communication and interpersonal skills, with the ability to translate complex technical concepts for non-technical stakeholders and act as a liaison to internal staff.
• Bachelors Degree in CS/Data Science

Responsibilities

• Use agentic coding to build agentic skills (third-party connections and tool-use logic).
• Troubleshoot and fix hallucination or logic errors in agent code when internal users are unable to train the agent via feedback.
• Work directly with internal employees to manage evaluations, guiding them on how to test the agent and translating their qualitative feedback into quantitative technical improvements.
• Provide feedback on how the existing agentic architecture handles memory, context, and tool-triggering.

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level