Senior Software Engineer, GenAI Evaluation

Google•New York, NY

3d•$174,000 - $252,000

About The Position

Google is seeking a software engineer to develop infrastructure for evaluating generative AI systems. The role involves building reusable evaluation infrastructure and improving the scalability, reliability, and efficiency of large-scale evaluation workflows across compute, storage, and serving backends. The engineer will work with product teams to onboard them, troubleshoot issues, and turn recurring needs into self-service features. The role also focuses on secure customer data isolation and access control, and helping teams progress from offline evaluations to online evaluations, live experiments, and production.

Requirements

Bachelor's degree or equivalent practical experience.
5 years of experience coding in Python.
3 years of experience developing large-scale infrastructure, distributed systems or networks, or working with compute technologies, storage, or hardware architecture.
3 years of experience working with generative AI agents.
2 years of experience with multithreading.

Nice To Haves

Master's degree or PhD in Computer Science or a related technical field.
2 years of experience with large-scale data processing, storage systems, or workflow orchestration.
2 years of experience working with two or more of the following: distributed and parallel systems, machine learning, information retrieval, natural language processing, Unix/Linux environments, or security and access control.
Experience with ML infrastructure, model evaluation, or LLM and GenAI workflows, such as autoraters, metrics, or experimentation platforms.
Experience developing in Python and C++.

Responsibilities

Design, build, and maintain reusable evaluation infrastructure, including streaming evaluation, autorater pipelines, and metric aggregation.
Improve the scalability, reliability, and efficiency of large-scale evaluation workflows across compute, storage, and serving backends.
Onboard product teams, debug their issues, and develop self-service features for recurring needs.
Build secure per-customer data isolation and access control for evaluation data and results.
Help teams move from offline evaluations to online evaluations, live experiments, and production.