South by Semantics: New horizons in evaluating pragmatic competence in language models

The South by Semantics Workshop at UT Austin is supported by funding from the departments of linguistics, philosophy, and computer science, and the College of Liberal Arts.
 

Abstract: For the first time in history, artificial models are using language like and with humans, sparking interest in whether LMs have learned the pragmatics of natural language. A dominant approach to evaluating pragmatics involves benchmarking how LMs capture standard phenomena such as implicature and figurative language. In this talk, I will explore approaches beyond this paradigm, using cognitive theories to interpret LM behaviors and highlight overlooked aspects of pragmatic competence in LMs. First, I will investigate “micro pragmatics” in LMs, showing that basic aspects of language (like when to use “the” and “that”) pose a pragmatic challenge even for internet-scale, Transformer-based LMs. Second, I will compare how humans and LMs make pragmatic inferences that are not linguistically mandated (“elicitures”) and arise through world modeling, in contrast to Gricean implicatures. Finally, on a methodological note, I will show how cognitive models can be used to interpret value tradeoffs in LMs’ utterance choices. These case studies highlight the relationship between pragmatics and world-building mechanisms, and suggest a blurry distinction between formal and functional linguistic competence.

Bio: Jennifer Hu is an Assistant Professor of Cognitive Science and Computer Science and member of the Data Science and AI Institute at Johns Hopkins University, where she directs the Group for Language and Intelligence. Her research aims to understand the computational principles underlying human language and communication. What representations and algorithms support language learning, comprehension, and production? How does language interact with other forms of knowledge and reasoning? And how can our understanding of human cognition help us develop better systems of artificial intelligence? Her lab approaches these questions by integrating computational modeling, behavioral experiments, and AI. Before joining Johns Hopkins, she worked with Tomer Ullman as a Research Fellow at the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University. She completed her PhD in Cognitive Science at MIT, where she was advised by Roger Levy.