Manager; AI Evaluation Engineering

Caterpillar Inc.•Peoria, IL

7d•$147,760 - $240,110

About The Position

Join the AI Engineering team of Cat Digital and lead a team dedicated to evaluating and validating generative AI solutions, including intelligent agents, digital assistants, and other capabilities. The role helps shape products that redefine how customers and dealers interact with technology through AI-driven experiences. The manager provides technical support and direction, oversees team and individual performance and development, owns the quality of AI engineering products, and establishes engineering best practices.

Requirements

Knowledge of major products and services and product and service groups, and ability to apply that knowledge appropriately to diverse situations.
Knowledge of software development tools and activities, and ability to produce software products or systems in line with product requirements.
Knowledge of the software development life cycle and ability to use a structured methodology to deliver and manage new or enhanced software products.
Knowledge of software quality assurance and testing, and ability to apply appropriate processes, tools, and techniques to assure high-quality software products and systems.

Nice To Haves

Proven experience leading software quality, test automation, validation, and AI evaluation teams, with the ability to scale processes, tools, metrics, and engineering practices across multiple products and enterprise initiatives.
Demonstrated success delivering enterprise-scale software and GenAI solutions across hybrid cloud and embedded/edge environments using modern engineering practices such as CI/CD, automated testing, incident management, feature flags, and progressive deployment strategies such as blue/green and canary deployments.
Deep expertise in GenAI architecture and system design, including LLMs, SLMs, multimodal, speech, and real-time models; prompt engineering; agentic systems; tool use and orchestration; RAG architectures; vector databases, embeddings, and chunking strategies; LoRA fine-tuning; and emerging standards such as MCP and A2A.
Hands-on experience with AI development and evaluation ecosystems, including Azure, AWS, GCP, Azure AI Foundry, SageMaker, Bedrock, Snowflake Cortex, LangChain, LangGraph, Langfuse, Arize, LangSmith, Humanloop, Ragas, DeepEval, Phoenix, or similar technologies.
Deep understanding of AI evaluation methodologies and challenges of non-deterministic systems, including metric design, RAG quality assessment, output reliability, safety evaluation, A/B testing, human-in-the-loop validation, and balancing deterministic and probabilistic testing approaches.
Expertise in AI observability and monitoring practices, including tracing, telemetry, evaluation data collection, root-cause analysis, and using AI for automated test generation, synthetic data creation, scenario expansion, and LLM-assisted evaluation.

Responsibilities

Provide technical support and direction so the team is aligned with company goals and capable of evaluating advanced AI projects.
Oversee individual and team performance and foster learning by identifying and addressing training and development needs.
Own the quality of AI engineering products, ensuring solutions are robust, reliable, and meet high standards.
Establish and supervise implementation of engineering best practices to maintain consistency and excellence in development processes.