AI Linguist

LinkedIn•Mountain View, CA

6d•Hybrid

About The Position

This role will be based in Mountain View, CA. At LinkedIn, our approach to flexible work is centered on trust and optimized for culture, connection, clarity, and the evolving needs of our business. The work location of this role is hybrid, meaning it will be performed both from home and from a LinkedIn office on select days, as determined by the business needs of the team. HALO (Human Judgment, Annotation, Localization, and Operations) is a horizontal team within Core AI that partners across the company to enable high-quality human judgment for AI development. We partner closely with cross-functional stakeholders and internal teams to define quality goals, design evaluation and data pipelines, and scale repeatable measurement systems. Our work spans multiple initiatives at once, supported by shared standards, platforms, and best practices that help teams move faster without compromising quality.

Requirements

Bachelor of Arts or Bachelor of Science in Computational Linguistics, Linguistics, Language Technologies, or related field
1+ year industry experience in human judgment, annotation, or evaluation supporting AI development
Experience creating rubrics or guidelines and producing labeled datasets or evaluation results used for AI training or evaluation
Experience operating scaled workflows using vendors or contractor workforces and annotation or evaluation platforms
Experience applying generative AI tools in data or evaluation workflows (prompt iteration, hybrid review, automated checks)
Python (or equivalent) for dataset quality assurance, metrics, sampling, and analysis (analysis scripting; not production software engineering)
Demonstrated ownership of small-to-medium projects from scope through delivery
Good written and verbal communication; able to document decisions, standards, and results clearly

Nice To Haves

Master of Science (or higher) in a relevant field
Experience building repeatable evaluation components (scorecards, regression sets, lightweight monitoring)
Familiarity with Jupyter and Git (or equivalent)
Experience evaluating multi-step or agentic behavior using scenario-based evaluations and failure mode tracking
Experience supporting multi-market evaluation consistency with localization or language experts
AI evaluation and annotation frameworks
Agentic AI
Quality governance and human-judgment operations
Cross-functional AI program execution

Responsibilities

Be part of working groups with Engineering, Product, Vendor Operations, etc. to align on quality goals, tradeoffs, and deliverables for assigned projects
Create annotation tasks end-to-end (rubrics, rating scales, decision rules, examples) and translate ambiguous behaviors into measurable criteria
Perform hands-on annotation and task completion for complex or high-impact cases to establish ground truth and calibrate guidelines
Build and operate scalable annotation and evaluation workflows on vendor or in-house platforms (task design, routing, sampling, quality assurance gates)
Support vendor and internal workforce execution through clear instructions, training materials, calibration support, and ongoing quality loops
Implement quality controls such as audits, inter-annotator agreement tracking, adjudication processes, and drift checks
Use AI tools to improve workflow speed and consistency (LLM-assisted prompting, hybrid labeling and evaluation, automated checks) with reliability controls
Create and validate datasets and evaluations used for training and assessing models and agents; perform error analysis and drive iteration with partners
Run practical experiments to improve quality or efficiency (rubric, prompt, task design, sampling changes) and document outcomes
Contribute requirements and feedback for workflow and tooling improvements