Sr. AI Engineer (Agentic AI & AWS Bedrock) Remote/Hybrid with Travel
Simple Solutions•Jacksonville, FL
•$70 - $95
About The Position
The Sr. AI Engineer will build, deploy, and monitor production-grade AI agents and generative AI solutions using AWS technologies. The role focuses on developing agents with Claude Agent SDK and Python, deploying them on Amazon Bedrock AgentCore Runtime, automating deployments with GitHub Actions, and using Langfuse for agent observability and continuous evaluation. It combines AI engineering, cloud deployment, data pipeline development, and LLM performance monitoring. The engineer will take agents from development through production deployment and ongoing evaluation, designing reliable, scalable, secure, and observable enterprise AI solutions. The role is remote or hybrid for candidates located in western U.S. time zones and requires comfort traveling to client sites for workshops and meetings.
Requirements
- Strong Python development experience.
- Hands-on experience with Claude Agent SDK, AI agent development, agent orchestration, and tool calling.
- Experience with Amazon Bedrock, Bedrock AgentCore Runtime, AWS Step Functions, Amazon Textract, and Amazon Comprehend.
- Experience deploying and supporting AI agents in production.
- Experience with Langfuse deployment, LLM tracing, monitoring, and performance analysis.
- Experience with continuous agent evaluation, LLM-as-a-Judge, evaluation metrics, and automated testing.
- Experience with GitHub Actions, CI/CD pipelines, Git, and deployment automation.
- Experience with vector databases, embeddings, chunking strategies, indexing, and knowledge base integrations for RAG.
- Understanding of AWS architecture, IAM, CloudWatch, security, and deployment practices.
- Strong understanding of AWS cloud services, LLM architectures, and automated deployment processes.
- Ability to independently design, implement, troubleshoot, and optimize solutions.
- The posting states overall experience of 12–20+ years, describes the role as 6–8+ years, and lists qualifications of 10–15+ years in software engineering, cloud engineering, or AI/ML development.
Nice To Haves
- Experience with LangChain, LangGraph, or other agent orchestration frameworks.
- Experience with RAGAS, DeepEval, or similar LLM evaluation frameworks.
- Familiarity with Amazon OpenSearch, Pinecone, FAISS, or pgvector.
- Experience with Docker, Terraform, or infrastructure as code.
- Knowledge of Model Context Protocol (MCP).
- Experience with prompt engineering, guardrails, and AI governance.
- Experience deploying enterprise generative AI solutions in production environments.
Responsibilities
- Design, develop, and deploy AI agents using Claude Agent SDK and Python.
- Build autonomous agents that execute complex workflows and interact with external tools, APIs, and knowledge bases.
- Develop agent orchestration workflows with error handling, retry mechanisms, and guardrails.
- Integrate AI agents with enterprise applications and data sources.
- Deploy and manage agents using Amazon Bedrock AgentCore Runtime.
- Develop scalable, secure, production-ready agent architectures on AWS and integrate agents with Bedrock foundation models and enterprise knowledge bases.
- Implement agent authentication, access controls, logging, and runtime monitoring.
- Deploy, configure, and maintain Langfuse for LLM and agent observability.
- Integrate Langfuse to capture traces, prompts, responses, tool calls, and execution metrics; monitor performance, latency, token consumption, and costs.
- Troubleshoot production issues using distributed tracing and monitoring dashboards.
- Design continuous evaluation frameworks for production agent traffic and evaluate accuracy, relevance, hallucinations, task completion, and tool execution quality.
- Develop automated evaluation pipelines using Langfuse, LLM-as-a-Judge, and custom metrics; implement feedback mechanisms, quality benchmarks, and regression detection.
- Develop AWS Step Functions workflows integrating Amazon Textract and Amazon Comprehend for intelligent document processing.
- Build document ingestion, preprocessing, and transformation pipelines; implement vector database solutions, including chunking, embeddings, and indexing.
- Integrate agents with RAG pipelines and enterprise knowledge bases, and optimize retrieval performance and contextual accuracy.
- Build and maintain GitHub Actions CI/CD pipelines to automate builds, tests, deployments, and configuration management.
- Automate deployments to Amazon Bedrock AgentCore Runtime and integrate agent evaluations into CI/CD.
- Apply DevOps best practices for version control, release management, and production monitoring.
- Independently design, implement, troubleshoot, and optimize AI agent solutions.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
Upload and Match Resume
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
