About The Position

Apple Services Engineering is building a scientific foundation for evaluating LLMs, agentic systems, and human-AI interaction. The team develops rigorous, scalable evaluation methods that help teams determine what to measure, whether a change helped, why a system failed, and whether it is ready to ship. The Applied Scientist will help build self-service evaluation tools, SDKs, and AI-powered capabilities that enable engineers across Apple to evaluate AI features independently, quickly, and correctly. The role develops methods that improve evaluation quality and help engineers interpret results and determine next steps. When an appropriate method does not exist, the scientist will extend existing research or develop new methods, then share findings through publications and internal research reports. Working with measurement scientists, ML researchers, platform engineers, and research engineers, the scientist will co-develop methods, ensure tools provide valid and trustworthy signals, and bring difficult problems into the team's research agenda. The role also defines the assumptions, failure modes, and validation tests needed for methods to work reliably without expert oversight and to be incorporated into evaluation SDKs and APIs used across Apple. The successful candidate applies scientific depth and practical judgment to method selection, experiment design, diagnosis on real-world data, and clear communication of what evaluation results do and do not support.

Requirements

  • Ph.D. in Computer Science, Machine Learning, Statistics, or a closely related field, or a Master's degree in one of these fields with at least 3 years of applied research experience.
  • Deep expertise in evaluating AI systems through research on evaluation methods, substantial hands-on experience evaluating LLMs or agentic systems in real products, or both.
  • Strong proficiency in Python and the modern ML ecosystem.
  • Demonstrated ability to implement complex methods from recent papers and run large-scale experiments.
  • Hands-on experience with large language models, such as fine-tuning, inference, prompting pipelines, or building LLM-powered applications.
  • Demonstrated research ability through peer-reviewed publications, preprints, or equivalent research output.
  • Track record of applying research to practical problems, including methods, prototypes, or tools adopted by other teams or used to inform product decisions.
  • Solid software engineering fundamentals for research code, including version control, testing, reproducibility, and maintainable code.
  • Excellent written and verbal communication skills, including the ability to explain complex results and limitations to diverse audiences.
  • Comfort with ambiguity and new problem spaces, including building solutions without an established playbook.

Nice To Haves

  • First-author publications at top-tier venues such as NeurIPS, ICML, ICLR, ACL, or EMNLP, especially on evaluation methodology.
  • Experience with LLM-as-judge approaches, including calibration, rubric design, or bias mitigation.
  • Experience with benchmark design and validation, including IRT or contamination detection.
  • Experience with human evaluation methodology, including protocol design or quality control.
  • Experience with experimentation and comparisons between system versions, including A/B testing or uncertainty quantification.
  • Experience evaluating agentic and multi-agent systems, including failure diagnosis or automated red-teaming.
  • Familiarity with theoretical foundations of evaluation, such as measurement theory, validity frameworks, statistical learning theory, calibration, reliability, decision theory, causal inference, or preference aggregation.
  • Experience with practical challenges of evaluating generative AI and agentic systems, including non-deterministic outputs, token economics, multi-turn agent trajectories, and tool use.
  • Experience building methods or tools used directly by non-experts, such as developer-facing ML tooling or self-service analytics.
  • Familiarity with evaluation tools and frameworks such as DeepEval, Ragas, TruLens, LangSmith, or agent tracing and observability tools.
  • Experience with distributed computing or cloud-native tools such as Ray, Spark, Docker, or Kubernetes for running evaluations at scale.
  • Experience with cross-disciplinary work, such as collaborating with social scientists, psychometricians, or domain experts.
  • Experience in fast-moving, early-stage teams and helping define technical direction from the ground up.

Responsibilities

  • Develop rigorous evaluation methods for LLMs, agentic systems, and human-AI interaction.
  • Build and improve self-service evaluation tools, SDKs, and AI-powered capabilities for engineers across Apple.
  • Raise the quality of evaluations engineers run and help them understand results and determine next steps.
  • Extend existing research or develop new methods when current approaches do not address a problem.
  • Share research findings through publications and internal research reports.
  • Co-develop methods with research and measurement scientists and ensure tools provide valid, trustworthy signals.
  • Identify difficult evaluation problems that should shape the team's research agenda.
  • Define methods' assumptions, failure modes, and validation tests in collaboration with platform and research engineers.
  • Design experiments to assess whether methods work and diagnose their failures on real-world data.
  • Explain evaluation results and their limitations to diverse audiences.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service