Product Content Engineer

Meta • Menlo Park, CA

6h • $162,000 - $227,000

About The Position

Product Content Engineering is a horizontal function supporting initiatives across Meta. It partners closely with product and technical teams to solve problems through model-centered solutions, set quality standards, and build frameworks that help AI-powered experiences work for people. This Content Engineer will partner with Meta’s agent tooling team to define model quality standards and target behaviors for Meta’s internal AI assistant, an automated tool intended to drive employee productivity across internal systems, data, and workflows.

The role focuses on evaluation: building frameworks, rubrics, and pipelines for assessing AI outputs. The engineer will assess model behavior and difficulty, identify shortcomings, and work across engineering, product, research, and data science to improve model performance. The position calls for rigorous evaluation, comfort with ambiguity, and strong editorial and analytical judgment.

Requirements

• Bachelor's degree in Computer Science, Data Science, Linguistics, or a related field, or equivalent practical experience.
• 5+ years of experience working collaboratively with product, engineering, design, and user research teams.
• 2+ years working with generative AI products, AI evaluation, prompt engineering, annotation, and/or labeling and analysis.
• Experience designing and implementing evaluation frameworks, annotation guidelines, or quality rubrics for agentic systems.
• Data analysis skills, including exploring data, identifying patterns, and producing actionable insights.
• Experience applying critical thinking to lead data-driven analyses that inform product or user experience decisions and communicating findings to executive leadership.
• Proven track record of cross-functional collaboration and delivering results amid evolving requirements and competing priorities.

Nice To Haves

• Experience establishing ground truth for text-based agentic workflows, including multi-step reasoning, tool use, and end-to-end task completion, and creating annotation guidelines and quality rubrics for repeatable evaluations.
• Experience adhering to and implementing responsible, ethical AI practices, such as risk assessment, bias mitigation, and quality and accuracy reviews.
• Familiarity with AI evaluation methods such as human evaluation, model-as-judge, A/B testing, or red-teaming.
• Experience building dashboards, scripts, or workflows that codify evaluation metrics.
• Ongoing AI skill development, such as prompt/context engineering or agent orchestration, and staying current with emerging AI technologies.
• Experience with Python, SQL, or other tools for data analysis and evaluation automation.
• Ability to integrate AI tools to optimize or redesign workflows and drive measurable impact, such as efficiency gains or quality improvements.

Responsibilities

• Define quality standards for model actions, task completion, state changes, and requests for permission, and use them to systematically evaluate AI models for Meta’s internal assistant.
• Design golden sets, taxonomies, and guidelines that integrate qualitative and quantitative signals.
• Build repeatable workflows for collecting, annotating, and analyzing AI outputs so evaluations can run efficiently as models evolve.
• Evaluate successive model releases through structured comparison, documenting improvements, regressions, and priorities.
• Develop processes to track model quality and performance over time and flag regressions.
• Synthesize evaluation results into structured error patterns and concrete recommendations for engineering and product teams.
• Work cross-functionally with engineering, data science, and product teams to align AI behaviors with real-world user expectations.
• Identify and mitigate operational risks, including staffing gaps, timeline conflicts, and quality regressions, before they affect delivery.

Benefits

• Bonus
• Equity
• Benefits

What This Job Offers

• Job Type: Full-time
• Career Level: Senior