CLiPS Colloquium - Kyle Mahowald: Learning About Language Models from a Lovely Three Weeks in Antwerp
Date: June 18 at 11:00
Location: Room S.R.118
Registration: confirm your attendance by sending an email to jens.lemmens@uantwerpen.be
Abstract: In recent years, language models have become adept at producing fluent English text, raising the possibility that they meaningfully learn something about grammatical structure. But questions remain as to the extent to which they are just memorizing from their huge training sets. While this is a very general question, I will attempt to address it by instead focusing on a very specific English grammatical construction: the Article+Adjective+Numeral+Noun AANN construction ("a lovely 2 days in Antwerp"). After first establishing that large language models match human judgments on specific constraints on this construction, I will report new results from a series of experiments where we train small language models on human-scale corpora (BabyLM: 100 million words), systematically manipulating the input corpus (e.g., removing all instances of AANNs from training, perturbing all AANNs to turn them into NAANs, removing all instances of related phenomena like "a few X" etc.) and pretraining models from scratch. Surprisingly, it seems like models can learn something about AANNs even without encountering any. I will argue that this experiment provides some evidence that models learn rare grammatical phenomena by shared structure with more frequent phenomena. If time remains, I will discuss evidence from an embedding projection method, which suggests that models also learn to give this construction a different, more measure-related construal.
Bio: Kyle Mahowald is an assistant professor in linguistics at the University of Texas at Austin (UT Austin), where he is a member of the Computational Linguistics Research Group. His research focuses on computational psycholinguistics, particularly on understanding how linguistic efficiency can explain aspects of linguistic behavior and typology. In 2016, he obtained his Ph.D., advised by Edward Gibson, at MIT’s Brain and Cognitive Sciences department. Kyle completed his postdoctoral research with Dan Jurafsky and Dan McFarland at Stanford, and previously worked as assistant professor in linguistics at the University of California, Santa Barbara.
Studiedag over Generatieve AI (Seminar on Generative AI)
Op vrijdag 21 juni co-organiseert Prof. Dr. Walter Daelemans samen met KANTL een studiedag over generative AI voor het Nederlands en de neerlandistiek. Raadpleeg de website van het evenement hier voor meer informatie.