Sr. AI Engineer (Agentic AI & AWS Bedrock) Remote/Hybrid with Travel

Simple Solutions • Jacksonville, FL

3d

About The Position

We are seeking an experienced AI Engineer to build, deploy, and monitor production-grade AI agents and Generative AI solutions using AWS technologies. The role focuses on agent development with Claude Agent SDK and Python, deployment on Amazon Bedrock AgentCore Runtime, GitHub Actions-based automation, and Langfuse observability and continuous evaluation.

The engineer will develop autonomous agents that execute complex workflows and interact with external tools, APIs, enterprise applications, data sources, and knowledge bases. The position also covers AWS architecture and security, document-processing and RAG pipelines, vector database integration, CI/CD, and ongoing monitoring and optimization of agent quality and operational performance.

The role is remote or hybrid for candidates located in western time zones, with travel to client sites for workshops and meetings. The posting lists 12–20+ years of overall experience and separately describes the AI Engineer experience level as 6–8+ years; qualifications also specify 10–15+ years of software, cloud, or AI/ML experience.

Requirements

• Strong Python development experience.
• Hands-on experience with Claude Agent SDK, AI agent development, agent orchestration, and tool calling.
• Experience with Amazon Bedrock, Bedrock AgentCore Runtime, AWS Step Functions, Amazon Textract, and Amazon Comprehend.
• Experience deploying, configuring, or maintaining Langfuse and working with LLM tracing, monitoring, and performance analysis.
• Experience with continuous agent evaluation, LLM-as-a-Judge, evaluation metrics, and automated testing.
• Experience with GitHub Actions, CI/CD pipelines, Git, and deployment automation.
• Experience with RAG, vector databases, embeddings, chunking strategies, indexing, and knowledge base integrations.
• Knowledge of AWS architecture, IAM, CloudWatch, security, and deployment practices.
• Strong hands-on experience building Generative AI or Agentic AI applications.
• Demonstrated experience deploying and supporting AI agents in production.
• Strong understanding of AWS cloud services, LLM architectures, and automated deployment processes.
• Overall experience is listed as 12–20+ years; the AI Engineer experience level is listed as 6–8+ years; qualifications specify 10–15+ years of software engineering, cloud engineering, or AI/ML development experience.
• Must be located in a western time zone.
• Must be comfortable traveling to client sites for workshops and meetings.

Nice To Haves

• Experience with LangChain, LangGraph, or other agent orchestration frameworks.
• Experience with RAGAS, DeepEval, or similar LLM evaluation frameworks.
• Familiarity with Amazon OpenSearch, Pinecone, FAISS, or pgvector.
• Experience with Docker, Terraform, or infrastructure as code.
• Knowledge of Model Context Protocol (MCP).
• Experience with prompt engineering, guardrails, and AI governance.
• Experience deploying enterprise Generative AI solutions in production environments.

Responsibilities

• Design, develop, and deploy AI agents using Claude Agent SDK and Python.
• Build autonomous agents that execute complex workflows and interact with external tools, APIs, and knowledge bases.
• Develop agent orchestration workflows with error handling, retries, and guardrails.
• Integrate AI agents with enterprise applications and data sources.
• Deploy and manage agents using Amazon Bedrock AgentCore Runtime.
• Develop scalable, secure, production-ready agent architectures on AWS and integrate agents with Amazon Bedrock foundation models and enterprise knowledge bases.
• Implement agent authentication, access controls, logging, and runtime monitoring.
• Deploy, configure, and maintain Langfuse for LLM and agent observability.
• Capture and monitor traces, prompts, responses, tool calls, execution metrics, response latency, token consumption, and operational costs.
• Troubleshoot production issues using distributed tracing and monitoring dashboards.
• Design and implement continuous evaluation frameworks for production agent traffic.
• Evaluate agent responses for accuracy, relevance, hallucinations, task completion, and tool execution quality.
• Build automated evaluation pipelines using Langfuse, LLM-as-a-Judge, and custom metrics; establish benchmarks and detect performance regressions.
• Implement continuous feedback mechanisms to improve agent reliability and effectiveness.
• Develop AWS Step Functions workflows integrating Amazon Textract and Amazon Comprehend for intelligent document processing.
• Build document ingestion, preprocessing, and data transformation pipelines.
• Implement vector database solutions, including document chunking, embedding generation, and vector indexing.
• Integrate agents with RAG pipelines and enterprise knowledge bases, and optimize retrieval performance and contextual accuracy.
• Build and maintain GitHub Actions CI/CD pipelines for agent builds, testing, deployment, and configuration management.
• Automate deployments to Amazon Bedrock AgentCore Runtime and integrate agent evaluations into CI/CD.
• Apply DevOps practices for version control, release management, and production monitoring.
• Independently design, implement, troubleshoot, and optimize AI agent solutions.

What This Job Offers

• Job Type: Full-time
• Career Level: Senior
• Education Level: No Education Listed