Hiring: AI Engineer | Agentic AI, AWS Bedrock AgentCore & Langfuse | 15+ Years

TMS LLC • Los Angeles, CA

7d • Hybrid

About The Position

We are seeking a hands-on AI Engineer to build, deploy, monitor, and continuously improve production-grade AI agents. The role focuses on Python, Agentic AI, AWS Bedrock AgentCore, and LLM evaluation. The position is based in Los Angeles, California, with a hybrid work arrangement. The engagement is listed as 3+ months, with a possible extension. The posting includes both a stated experience range of 15+ years in its heading and a qualification of 6–8+ years of relevant experience.

Requirements

• Python and Agentic AI development experience.
• Experience with AWS Bedrock AgentCore Evaluations or Bedrock evaluation.
• Experience designing and curating golden datasets.
• Experience designing and calibrating LLM-as-a-Judge evaluations against human labels.
• Experience integrating automated evaluations and CI gates.
• Experience with Langfuse and/or OpenTelemetry.
• 6–8+ years of software engineering, cloud engineering, or AI/ML experience.
• Hands-on experience with production AI agents and continuous evaluation.
• Strong problem-solving and troubleshooting skills.
• The posting heading states 15+ years of experience.

Nice To Haves

• RAGAS or equivalent evaluation frameworks.
• Trajectory evaluation and agentic red-teaming.
• Bedrock Guardrails, Strands Agents SDK, and GitHub Actions.

Responsibilities

• Develop AI agents using Python and Claude Agent SDK.
• Deploy agents using AWS Bedrock AgentCore Runtime.
• Design golden datasets and implement LLM-as-a-Judge evaluations.
• Integrate automated evaluations into CI/CD pipelines as quality gates.
• Implement Langfuse and/or OpenTelemetry for AI observability and tracing.
• Evaluate agent accuracy, task completion, tool execution, and performance.
• Support RAG pipelines and AWS data processing workflows.

What This Job Offers

• Career Level: Senior
• Education Level: No Education Listed