Research Engineer, Benchmarks & Evaluation

Ramp•New York, NY

2d•$204,400 - $352,000

About The Position

Ramp Labs is Ramp’s AI research group. It post-trains models on finance work, builds the environments and evaluations used for learning, and ships those models into Ramp products. The Research Engineer will own Ramp’s benchmarking effort, turning real finance work—including expense policy review, invoice coding, month-end close, and spreadsheet Q&A—into benchmarks and reinforcement-learning environments. The role includes building graders and determining what is actually improving across models and agents. The benchmarks inform model routing, post-training decisions, and whether agents are ready to perform consequential financial tasks without human oversight. The work spans long-horizon agents operating across tools and systems, tool use, multi-step reasoning, and financial judgment. The role also connects evaluations to post-training, identifies reward hacking and contamination, and may produce external benchmarks and publications.

Requirements

Hands-on experience building LLM benchmarks or evaluations that people used to make real decisions.
Strong Python and software engineering skills, with the ability to produce reproducible and fast evaluation code.
Statistical judgment, including variance, confidence intervals, sample size, inter-rater agreement, and distinguishing meaningful changes from noise.
Working knowledge of post-training, including supervised fine-tuning and reinforcement learning with verifiable rewards, and how evaluations feed into it.
Comfort with SQL and large datasets, including working directly from production traces.
Good judgment about what is worth measuring.

Nice To Haves

Industry experience on an evaluation, benchmarking, or post-training team.
Experience building reinforcement-learning environments or verifiers for agentic tasks.
Published benchmark or evaluation research, such as NeurIPS Datasets & Benchmarks, ICML, or ACL work.
Experience building or contributing to evaluation frameworks such as verifiers, lm-evaluation-harness, or HELM.
Familiarity with accounting, finance, or other expert-domain work.
Public work available for review, such as a benchmark, evaluation suite, or failure-analysis write-up.

Responsibilities

Turn agent traces, customer edge cases, and expert labels into versioned benchmark suites for long-horizon agents, tool use, multi-step reasoning, and financial judgment.
Own the benchmarking effort by deciding what to measure, running benchmark suites across models and agent versions, and reporting results used for team decisions.
Design graders using rubrics, programmatic checks, and LLM-as-judge methods; calibrate them against human agreement and identify their limitations.
Turn evaluations into reward functions and reinforcement-learning environments, and catch reward hacking and contamination before release.
Publish external benchmarks and write-ups describing model performance on real finance work.

Benefits

Flexible PTO.
Centralized home-office equipment ordering.
Health and wellness stipend.
Budget for intra-office travel.
Weekly coffee stipend.
United States: 100% medical, dental, and vision insurance coverage for the employee, with partial coverage for dependents.
United States: One Medical annual membership.
United States: 401(k) with employer match on contributions made while employed by Ramp.
United States: Fertility HRA up to $10,000 per year.
United States: Parental leave of up to 16 weeks for birthing and bonding or 8 weeks for bonding only at 100% pay.
United States: Pet insurance.
United States: In-office lunch, snacks, drinks, and other perks.
United States: Relocation expense coverage to NYC or SF, if needed.
Canada: Group medical, dental, and vision coverage through Sun Life.
Canada: Life, AD&D, and disability coverage.
Canada: Fertility drug coverage up to $4,000 lifetime.
Canada: Group Retirement Plan with employer match (RRSP + DPSP).
Canada: Parental leave of up to 16 weeks for birthing and bonding or 8 weeks for bonding only at 100% pay, with additional time available at reduced pay.
Canada: Employee Assistance Program and virtual care through Lumino Health.
United Kingdom: Private medical insurance through Freedom Elite.
United Kingdom: Virtual GP and at-home care via eMed x Livi.
United Kingdom: Workplace pension through Penfold, with a salary sacrifice option.
United Kingdom: Parental leave of up to 16 weeks for birthing and bonding or 8 weeks for bonding only at 100% pay, with additional time available at reduced pay.