FriSem

Hyunwoo Gu, Ph.D., 2nd-year PhD student in Associate Professor Justin Gardner's lab, Department of Psychology, Stanford University 

Title: Balancing semantics and salience in visual search

Abstract: In everyday life, we often search for a semantically defined target, such as when asked to shop for "something to drink made out of fruit" in a supermarket. During this semantic search, sometimes non-target objects that are semantically related (e.g., oranges) may guide our search, whereas irrelevant objects that are salient to the sensory system (e.g., promotion banners) can also attract attention. In my talk, I will address how people balance these two driving forces of attention - semantics and salience - by analyzing a database of human eye movements during visual search in naturalistic images. We begin by isolating the semantic contributions to visual search by constructing an image-computable framework based on the "CLIP" model that exclusively utilizes the match between the linguistic and the visual representations of the target and the scene, respectively. Next, we show that while this semantic matching model alone could explain a considerable portion of search behavior, weighting the model with salience-based models such as "DeepGaze-II" could achieve a better prediction. I will conclude by discussing the implications of these results for potential future projects.