Lead Engineer, AI Platform
About The Position
Circle is hiring a Lead Engineer to build the evaluation frameworks, observability tooling, and diagnostic infrastructure for its AI-powered features. The AI Quality engineering team measures, diagnoses, and improves production AI systems, rather than conducting ML research or model training. The role focuses on ensuring AI agents work reliably, efficiently, and at scale. This is a hands-on player-coach position. The Lead Engineer will set technical direction, manage day-to-day priorities, and lead and grow the AI Quality engineering team while contributing directly to code. The role partners closely with AI Core engineering to assess whether product changes improve quality. Circle is a fully remote company with an async, documented, globally distributed team.
Requirements
- At least 7 years of experience building and shipping production software.
- Experience with LLM-powered agents that take real actions in a product, preferably involving complex tool use, planning or orchestration, and sub-agents.
- Ability to explain a shipped system and how its performance was evaluated.
- Comfort with Ruby on Rails and/or Python, or ability to learn them quickly; Python proficiency is a strong plus.
- Experience building evaluation or observability infrastructure for ML/AI systems, such as evaluation pipelines, scorers, dashboards, or CI/CD for evaluations.
- Experience with evaluation frameworks such as Braintrust, LangSmith, or similar.
- Experience designing datasets, annotation workflows, or labeling pipelines for ML/AI evaluation.
- Ability to learn quickly, experiment extensively, and work effectively in a dynamic, ambiguous, fast-paced environment.
- Strong alignment with Circle's company values.
- English proficiency at CEFR Level C2 / ILR Level 5 in speaking, writing, and reading.
Nice To Haves
- Python proficiency, especially for data and evaluation work.
- Experience evaluating complex, tool-using AI systems with multiple tools, planning or orchestration, and sub-agents.
Responsibilities
- Build and own evaluation infrastructure, including CI/CD pipelines, scorers, and datasets for assessing AI agents from individual tool calls through multi-turn conversations.
- Diagnose quality failures across agent pipelines, including planning, execution, tool selection, and tool trajectories; prototype solutions or identify priorities for AI Core engineering.
- Develop datasets for evaluation by establishing annotation workflows and creating AI-generated and simulated conversations.
- Run structured experiments across prompts, models, and the agent harness; evaluate new and open-source models against the production baseline and create a framework for model-change decisions.
- Improve cost and latency through model swaps, caching, and routing based on plan complexity.
- Lead and grow the AI Quality engineering team, set technical direction, manage day-to-day priorities, and remain hands-on in the code.
- Partner with AI Core engineering to provide confidence that changes to AI products improve quality.
Benefits
- U.S.-benchmarked compensation globally.
- Company equity with ongoing refresh grants.
- 35 days of paid time off each year.
- Remote-only work with company retreats twice a year.
- Flexibility and autonomy.
- Benefits supporting health, wellbeing, and professional growth.
Stand Out From the Crowd
Upload your resume and get instant feedback on how well it matches this job.
What This Job Offers
Job Type
Full-time
Career Level
Senior
Education Level
No Education Listed
