About The Position
We are seeking an experienced AI Engineer to build, deploy, and monitor production-grade AI agents and generative AI solutions using AWS technologies. The role involves developing agents with Claude Agent SDK and Python, deploying them on Amazon Bedrock AgentCore Runtime, automating deployments through GitHub Actions, and using Langfuse for observability and continuous evaluation. It combines AI engineering, cloud deployment, data pipeline development, and LLM performance monitoring. The engineer will design agentic workflows, connect agents to tools, APIs, enterprise applications, data sources, and knowledge bases, and build secure and scalable AWS architectures. The role also includes implementing observability and evaluation systems, developing intelligent document-processing and RAG pipelines, and automating builds, tests, deployments, and configuration management. The position is based in Los Angeles, California, and is hybrid. The stated duration is three or more months, with a possible extension.
Requirements
- 6–8 or more years of software engineering, cloud engineering, or AI/ML development experience.
- Strong Python development experience.
- Hands-on experience building generative AI or agentic AI applications.
- Demonstrated experience deploying and supporting AI agents in production.
- Strong understanding of AWS cloud services, LLM architectures, and automated deployment processes.
- Experience with Claude Agent SDK, AI agent development, agent orchestration, and tool calling.
- Experience with Amazon Bedrock, Bedrock AgentCore Runtime, Step Functions, Textract, and Comprehend.
- Experience with Langfuse deployment, LLM tracing, monitoring, and performance analysis.
- Experience with continuous agent evaluation, LLM-as-a-Judge, evaluation metrics, and automated testing.
- Experience with GitHub Actions, CI/CD pipelines, Git, and deployment automation.
- Knowledge of vector databases, embeddings, chunking strategies, indexing, and knowledge base integrations.
- Knowledge of AWS architecture, IAM, CloudWatch, security, and deployment practices.
- Ability to build, deploy, and support reliable, scalable, observable enterprise AI solutions.
Nice To Haves
- Experience with LangChain, LangGraph, or other agent orchestration frameworks.
- Experience with RAGAS, DeepEval, or similar LLM evaluation frameworks.
- Familiarity with Amazon OpenSearch, Pinecone, FAISS, or pgvector.
- Experience with Docker, Terraform, or infrastructure as code.
- Knowledge of Model Context Protocol (MCP).
- Experience with prompt engineering, guardrails, and AI governance.
- Experience deploying enterprise generative AI solutions in production environments.
Responsibilities
- Design, develop, and deploy AI agents using Claude Agent SDK and Python.
- Build autonomous agents that execute complex workflows and interact with external tools, APIs, and knowledge bases.
- Develop agent orchestration workflows with error handling, retry mechanisms, and guardrails.
- Integrate AI agents with enterprise applications and data sources.
- Deploy and manage AI agents using Amazon Bedrock AgentCore Runtime.
- Develop scalable, secure, production-ready agent architectures on AWS.
- Integrate agents with Amazon Bedrock foundation models and enterprise knowledge bases.
- Implement agent authentication, access controls, logging, and runtime monitoring.
- Deploy, configure, and maintain Langfuse for LLM and agent observability.
- Integrate Langfuse with agents to capture traces, prompts, responses, tool calls, and execution metrics.
- Monitor agent performance, response latency, token consumption, and operational costs.
- Troubleshoot production issues using distributed tracing and monitoring dashboards.
- Design and implement continuous evaluation frameworks for production AI agent traffic.
- Evaluate agent responses for accuracy, relevance, hallucinations, task completion, and tool execution quality.
- Develop automated evaluation pipelines using Langfuse, LLM-as-a-Judge, and custom metrics.
- Implement feedback mechanisms to improve agent reliability and effectiveness.
- Establish quality benchmarks and detect agent performance regressions.
- Develop AWS Step Functions workflows integrating Amazon Textract and Amazon Comprehend for intelligent document processing.
- Build document ingestion, preprocessing, and data transformation pipelines.
- Implement vector database solutions, including document chunking, embedding generation, and vector indexing.
- Integrate agents with RAG pipelines and enterprise knowledge bases, and optimize retrieval performance and contextual accuracy.
- Build and maintain CI/CD pipelines using GitHub Actions.
- Automate agent code builds, testing, deployment, and configuration management.
- Implement deployment workflows for Amazon Bedrock AgentCore Runtime.
- Integrate automated agent evaluations into CI/CD processes.
- Apply DevOps best practices for version control, release management, and production monitoring.
- Independently design, implement, troubleshoot, and optimize AI agent solutions.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Career Level
Senior
Education Level
No Education Listed
