PCE Evals Lead

Meta•New York, NY

About The Position

A role serving as a key horizontal driver on evals efforts across teams – leading sprints, devising and executing automation strategies for manual efforts, and coordinating resources with PMs. This leader will be expected to take on highest priority work and flex as we continue to transform the team and uplevel the AI capabilities at all levels. Meta’s Product Content Engineering team is responsible for the content experiences across platforms and products. PCE works in close partnership with engineering, product, and partnerships teams to improve the discovery and distribution of content to drive value for users at scale. The work is time-sensitive and innovative, situated at the intersection of content expertise and Meta’s industry-leading ranking and delivery systems. The main function of a content engineer is to define and execute a content strategy that aims to significantly improve the product experience for consumers. The ideal Content Engineer has technical expertise combined with a capacity to drive change in ambiguous, fast-paced and high growth environments. They will be expected to perform data analytical work, comprehend complex recommendation systems, and test hypotheses while driving alignment across a wide cross-functional team to deliver business impact at scale.

Requirements

8+ years of demonstrated experience driving relevance on content platforms, with experience with video creators, ranking systems, content strategy, content labeling and analysis
1+ years working with GenAI products, prompt engineering, annotation, and/or content labeling and analysis
Expertise in developing and implementing high-quality datasets, annotations, and evaluation metrics that drive the training and improvement of generation models
Experience working independently and autonomously
Previous experience collaborating with product and technical teams
Proven knowledge of industry trends and news
Experience leading and collaborating on cross-functional projects to build products, respond to urgent needs, preempt issues and address concerns in collaborative and timely fashion
The Content Engineer should be experienced in content production and understanding, strategy and analysis to execute across a variety of projects, ideally in multiple languages

Nice To Haves

5+ years of work experience in product management, data analytics, or content operations in a leading content platform
2+ years of experience working closely with ranking teams
Experience in data visualization and tools such as Tableau

Responsibilities

Develop and execute strategies for FoA product teams to evaluate LLM models at scale across content work
Lead sprints for LLM model evaluations and auto judge creation in core PCE areas, while devising and implementing systems to increase evals capacity in areas involving the building of AI creator tools, content quality, content understanding and Search
Understand technical product needs and translate those into actionable direction for content-focused cross-functional teams
Understand Meta’s strategic and competitive position and deliver content strategies and product solutions that are aligned with our mission and recognized best in the industry
Define emergent opportunities and drive impact via execution in highly ambiguous areas. Agile support and flexibility is required