FriSem

Atlas Kazemian, PhD student in Psychology advised by Dr. Laura Gwilliams and Dr. Dan Yamins

Title: A unified framework for comparing computational models of spoken language understanding

Abstract: Recent advances in deep learning have led to studies using speech and language models to model neural responses to speech in the human brain. So far, these studies have largely focused on specialized domains, each targeting isolated modalities (speech or text). While fruitful, there lacks a comprehensive comparison of different speech and language models in their neural predictivity across brain regions. Using intracranial EEG data during naturalistic story listening, we examine the brain’s end-to-end processing of spoken language within a single framework using models spanning acoustic, speech, and language representations. Our findings are twofold. First, we see an alignment between speech-to-language models, recapitulating the known anatomical hierarchy of the brain. Second, we find that specific models are better than others at predicting speech and language responses in the brain. We close by discussing a new direction involving the development of an end-to-end audio–speech model for semantic understanding.