Colloquium

Melanie Mitchell, Professor at the Santa Fe Institute and inaugural Fractal Faculty member

Title: Investigating Abstract Reasoning in Humans and Machines

Abstract: State-of-the-art AI models have matched or exceeded human performance on reasoning benchmarks such as the Abstraction and Reasoning Corpus, prominent benchmark for conceptual understanding and analogy. But does high accuracy on this benchmark mean that these models understand and reason with the humanlike abstractions intended by the task creators? In this talk I will describe an evaluation methodology, inspired by experimental methods in cognitive science, to assess and compare the abstraction abilities of AI "reasoning" models an human participants. Our evaluations show that, while some models match or exceed human accuracy, their reasoning is frequently based on surface-level patterns or spurious associations, and thus lacks generalizability.  I will speculate on what is still needed to capture humanlike abstract reasoning in AI models.