Applied Research Scientist, LLM Evaluation & Post-Training

Innodata Inc.

2d•$175,000 - $225,000

About The Position

Innodata is expanding its GenAI research capability to advance state-of-the-art evaluation and post-training methods for LLM and multimodal systems. As an Applied Research Scientist, LLM Evaluation & Post-Training, you will lead research and experimentation on how evaluation design, measurement strategies, and feedback signals influence model improvement. This role is ideal for a technically rigorous researcher who is deeply fluent in modern LLM evaluation and post-training, and who can turn research insight into practical methods for customer solutions and internal platform innovation. You will work across human-in-the-loop and AI-augmented workflows, partnering with Language Data Scientists and AI/ML Research Engineers to design and validate evaluation frameworks that drive measurable model gains. The ideal candidate combines strong experimental and statistical judgment with hands-on technical ability and can engage as a peer with research and engineering stakeholders at leading AI companies.

Requirements

MS/PhD in Computer Science, Machine Learning, Statistics, Applied Mathematics, AI, or a related quantitative scientific field (PhD strongly preferred)
5+ years of relevant experience in applied research / research science in ML/AI, with substantial work in LLMs or foundation models
Demonstrated experience with LLM evaluation, benchmarking, alignment, post-training, or model quality research
Strong foundation in experimental design, statistical analysis, and scientific reasoning for ML systems
Strong coding skills in Python for research experimentation and analysis (e.g., data processing, evaluation pipelines, statistical analysis, visualization)
Experience working with modern ML tooling/frameworks (e.g., PyTorch, Hugging Face, JAX/TensorFlow as applicable) sufficient to design and execute model/evaluation experiments
Ability to evaluate and compare human and automated evaluation methods, including tradeoffs in cost, reliability, validity, and scalability
Experience designing evaluation studies and protocols that are reproducible across datasets, model versions, and evaluation runs
Ability to collaborate directly with technical stakeholders including research scientists, ML engineers, data scientists, and customer technical counterparts
Strong communication skills and ability to present nuanced technical conclusions, assumptions, and limitations clearly

Responsibilities

Define and execute a research agenda focused on LLM evaluation and post-training, especially evaluation-driven model improvement
Design rigorous experiments to study how evaluation methodologies impact fine-tuning and post-training outcomes
Develop and validate evaluation frameworks for LLM and multimodal systems, including benchmark/task design, scoring methods, judge/model-assisted evaluation, human evaluation protocols, and robustness/stress testing
Lead research on advanced evaluation domains, including long-context, cross-modal, and dynamic multi-turn evaluations
Study the effectiveness and limitations of existing evaluation techniques, and propose improved methodologies with clear validity and scalability tradeoffs
Analyze model behavior and failure patterns; generate actionable recommendations for model improvement and evaluation redesign
Collaborate with AI/ML Research Engineers to translate research methods into scalable evaluation and post-training pipelines
Collaborate with Language Data Scientists to integrate human-in-the-loop and synthetic data/evaluation strategies into research programs
Engage with customer technical stakeholders to understand evaluation goals, review methodologies, and provide expert recommendations
Contribute to internal benchmark datasets, evaluation frameworks, and reusable research assets
Produce high-quality technical documentation, internal research reports, and client-facing materials explaining methods, results, assumptions, and limitations
Contribute to thought leadership and best practices in LLM evaluation, post-training, and GenAI quality measurement