Senior Applied AI Engineer

Handshake•San Francisco, CA
•$220,000 - $260,000

About The Position

Handshake AI is building an evaluations platform to measure and improve the quality of expert human and synthetic data used to train frontier AI models. The platform aims to help AI labs determine whether data is accurate, improves model performance, and is worth the cost of producing it. The Senior Applied AI Engineer will own work from problem definition through production delivery, collaborating with researchers, customers, operators, and AI labs. The role focuses on building scalable evaluation products and production agents, creating reliable ways to measure data quality and model performance, and iterating through experiments to improve outcomes. The engineer will also provide technical direction, mentor engineers, and contribute to technical excellence across the engineering organization.

Requirements

  • At least 5 years of engineering experience.
  • Substantial experience shipping and operating high-quality software, owning decisions across the stack, and improving products based on actual use.
  • Experience building production agents and knowledge of tool calling, context management, orchestration, retrieval, human review, and reliability tradeoffs.
  • Excellent Python skills and ability to build scalable services, APIs, and data pipelines, with disciplined testing, debugging, and system design.
  • Ability to choose meaningful metrics, build representative test sets, assess graders, distinguish genuine improvements from noise, and iterate based on results.
  • Understanding of modern models, agent harnesses, benchmarks, evaluation techniques, and how training data and post-training methods influence model behavior.
  • Sound technical judgment and ownership, including the ability to design systems, make tradeoffs, and turn unclear problems into dependable solutions in a fast-moving environment.
  • Collaborative, low-ego communication and ability to work effectively with engineers, researchers, domain experts, operators, and customers.
  • Demonstrated ability through shipped products, open-source contributions, benchmarks, datasets, or other substantive technical work.

Nice To Haves

  • Research or applied experience in reinforcement learning, alignment, model behavior, synthetic data, or human-in-the-loop systems.
  • Experience developing LLM or agent benchmarks, reward models, evaluation methodologies, annotation systems, or data-quality frameworks.

Responsibilities

  • Build an evaluations platform by translating ambiguous quality needs into measurable outcomes and reusable products with operators, researchers, and labs.
  • Design and deploy production agentic workflows for generating, reviewing, investigating, and improving data, including reliable tool use, orchestration, human oversight, and failure recovery.
  • Build graders, programmatic verifiers, and reward signals calibrated against expert judgment across LLMs, agents, multimodal models, and RL environments.
  • Develop evaluation datasets, benchmarks, and agent harnesses; analyze traces, diagnose failure modes, and run experiments to improve performance.
  • Own projects from concept through rollout in collaboration with product, design, operations, and research.
  • Set technical direction, mentor engineers, and drive technical excellence for the team and engineering organization.

Benefits

  • Equity in a fast-growing company.
  • 401(k) match.
  • Competitive compensation.
  • Financial coaching.
  • Medical coverage.
  • Dental coverage.
  • Vision coverage.
  • Mental health support.
  • $500 wellness stipend.
  • Paid parental leave.
  • Fertility benefits.
  • Parental coaching.
  • $2,000 annual learning stipend.
  • Flexible PTO.
  • 15 holidays and 2 flex days.
  • Internet support.
  • Commuting support.
  • Free lunch and gym access in the San Francisco office.
  • Team outings.
  • Referral bonuses.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service