FriSem

Jerome Han, 2nd-year Ph.D. student in Psychology, advised by Professor Jay McClelland

Presenting his First-Year Project (FYP)

Title: LLMs as models of language-based learning and generalization in humans

Abstract: With limited linguistic input, humans are able to rapidly learn and generalize about new information in the world. To what extent can learning mechanisms in large language models (LLMs) achieve the same, and thus serve as algorithmic hypotheses for this ability? In this work we introduce a simple task designed to compare human and LLM learning and logical generalization in a language-based knowledge acquisition setting. We find that while humans are able to systematically generalize about new knowledge acquired from linguistic input, LLMs that learn the same inputs using finetuning or in-context learning struggle to generalize and are far more influenced by divergent prior expectations than humans are. We then consider how humans approach our task through the lens of reasoning and representation, and we find that human cognitive strategies for our task are reflected in both their self-reflections and their behavioral traces. Finally, we find that a perfect retrieval augmented generation setting allows LLMs to capture the broad principles of human behavior, and we consider the implications of this for building cognitive models that capture the human ability to use language to learn efficiently about the world.