AI Agent Engineer

Scout Motors • Charlotte, NC

14h • Onsite

About The Position

Scout Motors is looking for an AI Agent Engineer to join their team. This role involves designing, developing, and deploying AI agents on Databricks, managing the full lifecycle from prototyping to production. The engineer will build and maintain evaluation frameworks using MLflow, drive continuous quality improvement through offline and online evaluation, and incorporate human feedback into agent improvement cycles. Collaboration with Data Engineers, IT, and R&D is key for defining data configurations and pipelines. The role also involves surfacing agent outputs and quality insights to business stakeholders through dashboards and reporting tools, and identifying and closing data quality gaps. This position is based out of Charlotte, NC, and requires 4-5 days per week in the office, with potential for occasional travel.

Requirements

• Bachelor's Degree in Computer Science, Software Engineering, Data Science, Artificial Intelligence, or a related field – or equivalent practical experience
• 4+ years of experience in MLOps, AI/ML engineering, or data engineering in production environments
• Experience designing and executing evaluation workflows for non-deterministic AI systems, including both offline (dataset-based) and online (production monitoring) evaluation strategies
• Solid understanding of LLM concepts relevant to agent quality: hallucination, grounding, retrieval quality, tool use reliability, safety
• Hands-on experience developing and deploying AI agents on Databricks (Mosaic AI / Databricks Agent Framework), including multi-agent architectures and agent lifecycle management
• Deep expertise in MLflow: Tracking, Model Registry, Deployments, Evaluation framework (built-in, guideline & custom judges) and Tracing for agent observability
• Strong proficiency in Python and agent development frameworks (e.g. LangChain, LlamaIndex, AutoGen, or similar)
• Proficiency in SQL and cloud-based storage systems (Delta Lake, Unity Catalog, or equivalent)
• Familiarity with CI/CD practices for ML systems (model versioning, automated testing pipelines, deployment gates)
• Strong communication skills – able to translate technical evaluation results into clear, business-relevant insights for quality and engineering stakeholders
• Structured problem-solving mindset with the ability to define methodologies, prioritize independently, and drive assignments to completion with minimal supervision
• Comfortable operating in multidisciplinary, fast-moving environments with ambiguous requirements
• Growth mindset – eager to stay current with the rapidly evolving AI agent and LLMOps landscape

Nice To Haves

• Experience with BI tools (e.g. Sigma, PowerBI)
• Experience building RAG pipelines including Vector Store integration (e.g. Databricks Vector Search), embedding models and chunking strategies
• Familiarity with Streamlit or Gradio for lightweight internal tooling
• Knowledge of responsible AI / AI governance frameworks

Responsibilities

• Design, develop and deploy AI agents on Databricks, owning the full lifecycle from prototyping to production – including versioning, monitoring, and maintenance of multiple agents running in parallel
• Build and maintain robust evaluation frameworks using MLflow's evaluation suite, implementing and customizing judges (built-in, guideline-based, and custom) to systematically assess agent quality across correctness, safety, and business-specific criteria
• Drive continuous quality improvement of AI agents through structured offline evaluation pipelines (curated datasets, benchmarks) and online production monitoring, leveraging MLflow tracing to understand agent execution patterns
• Collect, structure and incorporate human feedback from diverse stakeholder groups (engineering, quality, business) into evaluation workflows and agent improvement cycles
• Collaborate with Data Engineers, IT, and R&D to define data configurations and pipelines that feed AI agents with reliable, high-quality inputs
• Leverage and combine multiple data sources (on-site, off-site, connected vehicle data) to build agents that identify trends, anomalies, and quality signals at scale
• Surface agent outputs and quality insights to Quality business stakeholders through dashboards and visual reporting tools, translating complex model behavior into understandable and actionable information
• Identify and close data quality gaps, ensuring that data feeding into agents is clean, consistent, and continuously improving through well-defined requirements and automated checks

Benefits

• Competitive insurance including: Medical, dental, vision and income protection plans
• 401(k) program with: An employer match and immediate vesting
• Generous Paid Time Off including: 20 days planned PTO, as accrued
• 40 hours of unplanned PTO and 14 company or floating holidays, annually
• Up to 16 weeks of paid parental leave for biological and adoptive parents of all genders
• Paid leave for circumstances related to bereavement, jury duty, voting time, or military leave

What This Job Offers

• Job Type: Full-time
• Career Level: Mid Level
• Education Level: Associate degree