Hiring | AI Engineer – Agent Evaluation Specialist | C2C/W2 | 14+ Years Experience | Remote
About The Position
The AI Engineer – Agent Evaluation Specialist will focus hands-on on evaluating LLM-powered agents rather than primarily building or developing them. Agent evaluation is the core requirement and the primary selection criterion. The role includes designing, implementing, and running LLM-based evaluation frameworks; collaborating with Business Analysts and stakeholders to create, validate, and maintain golden datasets; and instrumenting agents with Langfuse to trace execution, capture evaluation scores, monitor performance, and troubleshoot failures. Evaluation spans agent behavior, tool usage, reasoning workflows, and response quality, including accuracy, relevance, and reliability. The role is listed as hybrid in Los Angeles, California, for a duration of three or more months, with a possible extension. C2C/W2 and an experience threshold of 14+ years are specified.
Requirements
- Hands-on experience designing, implementing, and running LLM-as-a-Judge evaluation frameworks.
- Experience developing and maintaining golden datasets in collaboration with Business Analysts and stakeholders.
- Practical experience instrumenting AI agents with Langfuse.
- Strong understanding of end-to-end agent evaluation, including behavior, tool usage, reasoning workflows, and response quality.
- Demonstrated implementation experience with agent evaluation frameworks, rather than only theoretical knowledge or general GenAI/LLM development.
- 14+ years of experience.
Responsibilities
- Design, implement, and run LLM-based evaluation frameworks to assess agent response quality, accuracy, relevance, and reliability.
- Collaborate with Business Analysts and stakeholders to create, validate, and maintain golden datasets for agent evaluation.
- Instrument AI agents using Langfuse to trace agent execution, capture evaluation scores, monitor performance, and troubleshoot failures.
- Evaluate agent behavior, tool usage, reasoning workflows, and response quality end to end.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Career Level
Senior
Education Level
No Education Listed
