AI Operations Engineer

ACA Group•Durham, NC

1d•Hybrid

About The Position

We are seeking an AI Operations Engineer to help support and optimize our next-generation AI platform focused on compliance, surveillance, and intelligent workflow automation. This role will work closely with AI engineers, product managers, and domain experts to evaluate AI outputs, manage training and evaluation datasets, improve prompts and agent behaviors, and help operationalize LLM-powered systems across the platform. The ideal candidate is detail-oriented, analytical, and excited about working hands-on with modern AI technologies including agents, MCP servers, vector search, RAG pipelines, document/media processing, and large language models.

Requirements

Detail-oriented
Analytical
Hands-on experience with modern AI technologies including agents, MCP servers, vector search, RAG pipelines, document/media processing, and large language models.

Responsibilities

Test and evaluate AI agents, prompts, and workflows across multiple use cases
Create, maintain, and curate labeled datasets for AI evaluation and tuning
Review AI-generated outputs for quality, accuracy, safety, and consistency
Update and optimize prompts, agent instructions, and AI “skills”
Help improve retrieval quality for vector search and RAG pipelines
Assist with evaluation frameworks and benchmark creation for LLM systems
Work with engineering teams to validate MCP connectors and integrations
Analyze model failures, hallucinations, and edge cases to improve system performance
Participate in human-in-the-loop review workflows for AI-generated content
Document testing procedures, findings, and recommended improvements
Collaborate with compliance and domain experts to refine AI behaviors and detection logic