FriSem

Daniel Wurgaft, Ph.D. Student in Psychology, admitted Autumn 2023, presenting his First-Year Project (FYP)

Title: Towards A Normative Theory Explaining The Emergence Of In-context Learning

Abstract: Large language models (LLMs) demonstrate a striking ability to learn without updates to their weights, a process known as in-context learning (ICL). This capacity enables LLMs to rapidly adapt to novel tasks, achieving generality that was previously unseen in neural networks. Despite its importance, the mechanisms behind ICL and its emergence during training remain poorly understood. Recent research has identified several factors influencing the emergence of ICL, including the necessity of diverse training tasks and the impact of task dimensionality and context length. In this work, we provide a Bayesian perspective on the training dynamics of ICL, offering early empirical evidence to explain these phenomena. Additionally, we introduce a novel experimental setting that replicates key ICL findings while allowing for clear analysis. We conclude by highlighting parallels to human meta-learning and suggesting directions for future research.