Master Thesis: Long Context Extension for Public Sector Document Retrieval

AI Sweden

2d • Hybrid

About The Position

As Sweden's national center for applied AI, AI Sweden is looking for a master thesis student to join their team. This thesis will focus on extending the context length of dense LLMs beyond their pre-training limits, which is essential for practical deployment in public administration. The project will investigate how efficiently Prelude 9B can be extended to 32k or 64k (or longer) tokens for Swedish administrative Retrieval-Augmented Generation (RAG) tasks without catastrophic forgetting of its core reasoning and language modeling capabilities. The goal is to extend and evaluate the context window of Prelude 9B for real-world utility, applying continued pre-training or Supervised Fine-Tuning (SFT) on a high-quality Swedish long-context dataset and evaluating the model in a localized RAG environment simulating Skatteverket's 'Väg 1' lab setting.

Requirements

• Ongoing Master’s studies in Computer Science, Data Science, Machine Learning, Engineering Physics, or a related quantitative field.
• Proficiency in Python and hands-on experience with modern deep learning frameworks (PyTorch, Hugging Face ecosystem).
• Familiarity with LLM post-training alignment (e.g., SFT, DPO, RLHF/RLVR) or context-extension, alongside comfort running distributed GPU training in Linux/HPC environments.
• Curious, self-driven MSc students eager to work at the frontier of open-weight European AI research (LLMs).
• Ability to thrive on empirical discovery, design rigorous experiments, and let data challenge assumptions.

Responsibilities

• Investigate RoPE scaling, YaRN, LongRoPE techniques, and Needle-in-a-Haystack (NIAH) evaluations.
• Apply continued pre-training or SFT on a high-quality Swedish long-context dataset to extend Prelude's window to 32k/64k tokens.
• Deploy the model in a localized RAG environment simulating Skatteverket's 'Väg 1' lab setting.
• Evaluate precision, recall, and MFU (Model FLOPs Utilization).

What This Job Offers

• Job Type: Full-time
• Career Level: Entry Level