Senior Research Engineer, AI Evaluation

System•New York, NY
•$180,000 - $240,000

About The Position

System is a Public Benefit Corporation developing software to help people see and solve problems as systems, beginning in healthcare. The company values autonomous, interdisciplinary, and diverse teams, agile development, continuous learning, and purpose-driven work. As Senior Research Engineer, you will lead the definition, implementation, and scaling of System’s LLM evaluation architecture and algorithmic benchmarking systems. You will also contribute to research and development supporting System’s core platform and product suite. This role reports to the Head of AI & Engineering. The position offers a base salary of $180,000 to $240,000, equity, and a comprehensive benefits package. System has a hub in Lower Manhattan and supports remote work. Applicants must be legally authorized to work in the United States and maintain ongoing work authorization during employment.

Requirements

  • At least 3 years of experience in software engineering, applied machine learning, or AI research.
  • A strong statistical foundation and scientific rigor.
  • Demonstrated experience designing evaluation harnesses or benchmarking suites for LLMs and complex reasoning systems, such as LLM-as-a-judge, reference-based metrics, or human-in-the-loop review.
  • Advanced proficiency in Python.
  • Experience with ML frameworks such as PyTorch and Hugging Face, frontier model APIs, RAG pipelines, and experiment-tracking or observability tooling.
  • Strong written communication skills, such as journal article writing, blog posts, or position papers.
  • Must be legally authorized to work in the United States and maintain ongoing work authorization during employment.

Nice To Haves

  • An advanced degree in Computer Science, Machine Learning, Statistics, or another scientific field.
  • Contributions to peer-reviewed research or presentations at machine learning conferences such as ICML, ICLR, or NeurIPS.
  • Experience with knowledge graphs.

Responsibilities

  • Design, build, and maintain automated, reproducible evaluation frameworks for grading LLM reasoning, retrieval accuracy, and structured outputs across clinical decision support applications.
  • Create general and domain-specific evaluation benchmarks, synthetic test-case generators, and golden datasets.
  • Quantify and mitigate edge cases, hallucinations, model drift, and safety regressions.
  • Collaborate with AI Research Scientists, ML Infrastructure Engineers, and Product teams to establish standardized offline and online evaluation loops integrated into CI/CD workflows.
  • Contribute to research and development supporting System’s core platform and product suite.

Benefits

  • Meaningful equity ownership.
  • Comprehensive health and wellness coverage designed to support employee well-being.
  • Generous paid time off.
  • Flexible workplace policy with a Lower Manhattan hub and full support for remote work.
© 2026 Teal Labs, Inc
Privacy PolicyTerms of Service