Technical Project Manager

LILT

20d

About The Position

We are looking for a Technical Project Manager to join our Applied AI team and lead large-scale multilingual data collection, human-in-the-loop workflows, and Large Language Model (LLM) evaluation initiatives. You will work with AI specialists, engineering teams, specialized Subject Matter Experts (SMEs), and data operations teams to translate technical AI requirements into high-throughput data workflows, hybrid evaluation pipelines, and measurable delivery plans. This role combines technical project management with hands-on data analysis, unit economics optimization, and a strong operational understanding of modern LLM training and evaluation processes.

Requirements

3–5+ years of technical project management experience in AI/ML, data platforms, or technical data operations.
Strong understanding of LLM training and evaluation concepts, including SFT, RLHF, human evaluation, automated evaluations, LLM-as-a-judge, and red teaming.
Proficiency in SQL for analyzing delivery velocity, data quality metrics, and cost structures.
Working knowledge of data pipelines, structured data formats, APIs, and validation processes.
Experience partnering with engineers to manage technical requirements, dependencies, and issue resolution.
Proven ability to track and optimize project unit economics (cost per token/task) and delivery velocity.
Strong communication skills, including the ability to translate technical requirements for multilingual audiences and diverse global contributor networks
Experience managing complex workflows using Agile, Scrum, or Kanban.

Nice To Haves

Experience using Python or scripting tools for data analysis, workflow automation, or evaluation scripting.
Experience managing high-skill Subject Matter Experts (SMEs) or specialized contributor networks across technical domains.
Experience with annotation platforms, automated evaluation tools, business intelligence tools, and Jira.
Experience delivering complex multilingual or multimodal data projects.
Background in computer science, data science, engineering, or equivalent practical experience.
Fluency in an additional language.

Responsibilities

Manage Applied AI data collection and evaluation projects from technical scoping through validation, delivery, and retrospective review.
Translate technical AI requirements into structured data specifications, domain-specific annotation requirements, evaluation plans, and acceptance criteria.
Manage timelines and throughput velocity, dependencies, delivery risks, and technical blockers across Applied AI, engineering, and operations stakeholders.
Coordinate data collection and annotation for supervised fine-tuning (SFT), preference data used in reinforcement learning from human feedback (RLHF), and model evaluation.
Partner with engineering teams to define and validate data workflows, including ingestion, annotation tooling, systems integrations, quality checks, data generation pipelines, and delivery.
Operationalize hybrid evaluation methodologies, combining human review with automated evaluation tooling, and LLM-as-a-judge frameworks.
Coordinate requirements for data formats, schemas, metadata, annotation tools, and system integrations.
Coordinate requirements for data formats, schemas, metadata, annotation tooling, and automated quality checks.
Work with Applied AI specialists to operationalize evaluation methods, including human review, response ranking, and safety testing.
Investigate data, pipeline, or tooling issues and coordinate prompt resolution with technical owners.
Use SQL and business intelligence tools to analyze throughput velocity, quality, unit economics, and supplier/delivery performance.
Define and monitor quality standards, including inter-annotator (IAA) agreement, gold-set performance, and dataset completeness.
Lead root-cause analysis on quality discrepancies, workflow bottlenecks, benchmark discrepancies, and quality edge cases.
Establish delivery validation checks and coordinate stakeholder acceptance of completed, accurate datasets, delivery targets, and evaluation results.
Translate complex technical requirements into actionable plans and clear instructions for global contributor teams, external vendors, and specialized domain SMEs and teams.
Communicate project status, technical risks and tradeoffs, unit economics, and operational risks to technical and non-technical stakeholders.
Ensure contributor feedback and evaluation findings inform improvements to internal tooling, guidelines, and dataset pipelines.