Generative AI Engineer
About The Position
We are seeking an experienced AI Engineer to build, deploy, and monitor production-grade AI agents and Generative AI solutions using AWS technologies. The role focuses on developing agents with the Claude Agent SDK, deploying them on Amazon Bedrock AgentCore Runtime, automating deployments with GitHub Actions, and implementing Langfuse for observability and continuous evaluation. It combines AI engineering, cloud deployment, data pipeline development, and LLM performance monitoring. The engineer will develop agent orchestration workflows; integrate agents with enterprise applications, data sources, APIs, tools, and knowledge bases; deploy secure, scalable agent architectures; and implement authentication, access controls, logging, and runtime monitoring. The role also includes building continuous evaluation frameworks for production agent traffic, monitoring quality and performance, and improving reliability through feedback and regression detection. Additional work includes developing document-processing and knowledge-base pipelines with AWS Step Functions, Amazon Textract, and Amazon Comprehend; implementing vector indexing and RAG workflows; and maintaining CI/CD pipelines for agent builds, tests, deployments, and evaluations.
Requirements
- 6–8+ years of software engineering, cloud engineering, or AI/ML development experience.
- Strong Python development experience.
- Strong hands-on experience building Generative AI or Agentic AI applications.
- Experience with Claude Agent SDK, AI agent development, agent orchestration, and tool calling.
- Experience with Amazon Bedrock, Bedrock AgentCore Runtime, AWS Step Functions, Amazon Textract, and Amazon Comprehend.
- Experience deploying and supporting AI agents in production.
- Experience deploying Langfuse and implementing LLM tracing, monitoring, and performance analysis.
- Experience with continuous agent evaluation, LLM-as-a-Judge, evaluation metrics, and automated testing.
- Experience with GitHub Actions, CI/CD pipelines, Git, and deployment automation.
- Experience with vector databases, embeddings, chunking strategies, indexing, and knowledge-base integrations.
- Strong understanding of AWS cloud services, LLM architectures, and automated deployment processes.
- Understanding of AWS architecture, IAM, CloudWatch, security, and deployment practices.
- Ability to independently design, implement, troubleshoot, and optimize AI agent solutions.
Nice To Haves
- Experience with LangChain, LangGraph, or other agent orchestration frameworks.
- Experience with RAGAS, DeepEval, or similar LLM evaluation frameworks.
- Familiarity with Amazon OpenSearch, Pinecone, FAISS, or pgvector.
- Experience with Docker, Terraform, or infrastructure as code.
- Knowledge of Model Context Protocol (MCP).
- Experience with prompt engineering, guardrails, and AI governance.
- Experience deploying enterprise Generative AI solutions in production environments.
Responsibilities
- Design, develop, and deploy AI agents using Claude Agent SDK and Python.
- Build autonomous agents that execute complex workflows and interact with external tools, APIs, and knowledge bases.
- Develop agent orchestration workflows with error handling, retry mechanisms, and guardrails.
- Integrate AI agents with enterprise applications and data sources.
- Deploy and manage AI agents using Amazon Bedrock AgentCore Runtime.
- Develop scalable, secure, production-ready agent architectures on AWS and integrate agents with Bedrock foundation models and enterprise knowledge bases.
- Implement agent authentication, access controls, logging, and runtime monitoring.
- Deploy, configure, and maintain Langfuse for LLM and agent observability.
- Integrate Langfuse to capture traces, prompts, responses, tool calls, and execution metrics; monitor performance, latency, token use, and costs.
- Troubleshoot production issues using distributed tracing and monitoring dashboards.
- Design and implement continuous evaluation frameworks for production AI agent traffic.
- Evaluate responses for accuracy, relevance, hallucinations, task completion, and tool execution quality.
- Build automated evaluation pipelines using Langfuse, LLM-as-a-Judge, and custom metrics.
- Implement feedback mechanisms, quality benchmarks, and regression detection to improve agent reliability and effectiveness.
- Develop AWS Step Functions workflows integrating Amazon Textract and Amazon Comprehend for intelligent document processing.
- Build document ingestion, preprocessing, and data transformation pipelines.
- Implement vector database solutions, including document chunking, embedding generation, and vector indexing.
- Integrate agents with RAG pipelines and enterprise knowledge bases; optimize retrieval performance and contextual accuracy.
- Build and maintain CI/CD pipelines using GitHub Actions.
- Automate agent code builds, testing, deployment, and configuration management, including deployment workflows for Bedrock AgentCore Runtime.
- Integrate automated agent evaluations into CI/CD and apply DevOps practices for version control, release management, and production monitoring.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Career Level
Senior
Education Level
No Education Listed
