Master Thesis: Calibrating LLM-as-a-Judge for Swedish Fluency vs. Adequacy

AI Sweden

2d • Hybrid

About The Position

As Sweden's national center for applied AI, AI Sweden is looking for a master thesis student to join their team. This project focuses on automated evaluation for tracking LLM post-training progress, specifically addressing the biases in "LLM-as-a-judge" paradigms, particularly for non-English languages like Swedish. The thesis will explore whether providing an LLM judge with few-shot, human-calibrated Swedish examples can effectively decouple fluency from adequacy, aligning automated Elo scores with human expert ratings. The goal is to refine and validate the JudgeArena framework for Swedish post-training evaluation.

Requirements

• Ongoing Master’s studies in Computer Science, Data Science, Machine Learning, Engineering Physics, or a related quantitative field.
• Proficiency in Python.
• Hands-on experience with modern deep learning frameworks (PyTorch, Hugging Face ecosystem).
• Familiarity with LLM post-training alignment (e.g., SFT, DPO, RLHF/RLVR) or context-extension.
• Comfort running distributed GPU training in Linux/HPC environments.

Nice To Haves

• Curiosity and self-driven attitude.
• Eagerness to work at the frontier of open-weight European AI research (LLMs).
• Thrive on empirical discovery, design rigorous experiments, and let data challenge assumptions.

Responsibilities

• Review automated evaluation paradigms, LLM-as-a-judge biases, and metrics distinguishing fluency from adequacy.
• Annotate a high-quality Swedish dataset with separate scores for fluency and adequacy.
• Use annotated data as calibration prompts for the Judge model in JudgeArena.
• Correlate the calibrated JudgeArena outputs against human judgments on a holdout set of Prelude 9B's generated responses.
• Quantify the improvement in Pearson/Spearman correlation.

Benefits

• Work alongside leading AI scientists and change leaders.
• Opportunity to drive research questions with long shelf-life and wide applicability to Swedish industry and the public sector.
• Aim for publications at competitive venues.
• Celebrate a culture of research excellence.
• Experience an organization positioned at the sweet spot of governmental influence and startup agility.
• Be in close contact with government, academia, and private and public sectors.

What This Job Offers

• Career Level: Entry Level