CLP21 Connectionist Language Processing (Summer 2021)

synopsis


This course will examine neurocomputational (or connectionist) models of human language processing. We will start from biological neurons, and show how their processing behaviour can be modelled mathematically. The resulting artificial neurons will then be wired together to form artificial neural networks, and we will discuss how such networks can be applied to build neurocomputational models of language learning and language processing. It will be shown that such models effectively all share the same computational principles, and that any differences in their behaviour is driven by differences in the representations that they process and construct. Near the end of the course, we will use the accumulated knowledge to construct a psychologically plausible neurocomputational model of incremental (word-by-word) language comprehension that constructs a rich utterance representation beyond a simple syntactic derivation or semantic formula.

course overview


Connectionist Language Processing is a course taught in the Department of Language Science and Technology at Saarland University. It is open for master-level students.

Lecturer: Harm Brouwer <me[at]hbrouwer.eu>
TA: Christoph Aurnhammer <aurnhammer[at]coli.uni-saarland.de>

Time: Tuesday 14:15-15:45; Thursday 14:15-15:45
Place: Online (Microsoft Teams)
Start: 20.04.21

Exam: Tuesday, July 20, 14:00-16.00
Credits: 6 CP

Registration: Send me an email to enrol for the course

Format and Requirements:

schedule


This is the provisional course schedule. See below for suggested background literature.

Date Topic
20.04.21 Lecture 1: Introduction to Connectionism and the Brain
22.04.21 Tutorial 1: Introduction to Neural Networks in MESH


27.04.21 Lecture 2: A Primer on Linear Algebra
29.04.21 Lecture 3: Learning in Single-layer Networks


04.05.21 Lecture 4: Training Multi-layer Networks
06.05.21 Tutorial 2: Training Multi-layer Networks


11.05.21 Lecture 5: Reading Aloud
13.05.21 (no class: Christi Himmelfahrt)


18.05.21 Tutorial 3: Reading Aloud
20.05.21 Lecture 6: English Past Tense


25.05.21 Tutorial 4: English Past Tense
27.05.21 Lecture 7: Simple Recurrent Networks I


01.06.21 Lecture 8: Simple Recurrent Networks II
03.06.21 (no class: Fronleichnam)


08.06.21 Lecture 9: Recurrent Neural Networks as Models of Sentence Processing (Christoph Aurnhammer)
10.06.21 Tutorial 5: Simple Recurrent Networks


15.06.21 Lecure 10: Modeling the Electrophysiology of Language Comprehension
17.06.21 Lecture 11: Situation Modeling using Microworlds


22.06.21 Tutorial 6: Expectation-based Comprehension I
24.06.21 (no class)


29.06.21 Lecture 12: Modeling the Neurobehavioral Correlates of Comprehension-centric Surprisal
01.07.21 Tutorial 7: Expectation-based Comprehension II


06.07.21 Lecture 13: Course Summary
08.07.21 Q&A


13.07.21 (no class)
15.07.21 (no class)


20.07.21 Exam

suggested literature


This is a inexhaustive list of suggested literature organized by lecture. For each lecture, the list is ordered in terms of the relevance/closeness of the articles to the material presented in that lecture. Articles marked with an asterisk (*) are (co-)authored by me.

20.04.21

  1. Plunkett K, and Elman J. (1997). Exercises in rethinking innateness: A Handbook for Connectionist Simulations. Cambridge, MA: MIT Press. Chapter 2.

27.04.21

  1. Jordan, M. I. (1986). An introduction to linear algebra in parallel distributed processing. In: Rumelhart, D. E., and McClelland, J. L., et al. Parallel Distributed Processing, Vol. 1, pp. 365-422.

29.04.21

  1. Plunkett K, and Elman J. (1997). Exercises in rethinking innateness: A Handbook for Connectionist Simulations. Cambridge, MA: MIT Press. Chapter 1.
  2. *Brouwer, H. (2014). The Electrophysiology of Language Comprehension: A Neurocomputational Model. PhD thesis. University of Groningen, Groningen, The Netherlands. Appendix A (up to and including A.2.2)

04.05.21

  1. Plunkett K, and Elman J. (1997). Exercises in rethinking innateness: A Handbook for Connectionist Simulations. Cambridge, MA: MIT Press. Chapters 1 and 4.
  2. *Brouwer, H. (2014). The Electrophysiology of Language Comprehension: A Neurocomputational Model. PhD thesis. University of Groningen, Groningen, The Netherlands. Appendix A (up to and including A.2.3)

11.05.21

  1. Plaut, D. C., McClelland, J. L., Seidenberg, M. S., and Patterson, K. (1996). Understanding normal and impaired word reading: Computational principles in quasi-regular domains. Psychological Review, 103, pp. 56-115. Pages 1-19 (up to and including Summary)

20.05.21

  1. Plunkett K, and Marchman V. A. (1991). U-shaped learning and frequency effects in a multi-layered perceptron: Implications for child language acquisition. Cognition, 38(1). pp. 43-102. Pages 1-15 (up to and including 3.1)

27.05.21

  1. Elman, J. (1990). Finding structure in time. Cognitive Science, 14, pp. 179-211.
  2. *Brouwer, H. (2014). The Electrophysiology of Language Comprehension: A Neurocomputational Model. PhD thesis. University of Groningen, Groningen, The Netherlands. Appendix A (up to and including A.2.4)

01.06.21

  1. Elman, J. (1991). Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 7, pp. 195-225.
  2. Elman, J. (1993). Learning and development in neural networks: the importance of starting small Cognition, 48, pp. 71-99.

15.06.21

  1. *Brouwer, H., Crocker, M. W., Venhuizen, N. J., and Hoeks, J. C. J. (2017). A Neurocomputational Model of the N400 and the P600 in Language Processing. Cognitive Science, 41(S6), pp. 1318-1352.

17.06.21

  1. *Venhuizen, N. J., Crocker, M. W., and Brouwer, H. (2019). Expectation-based Comprehension: Modeling the interaction of world knowledge and linguistic experience. Discourse Processes, 56:3, pp. 229-255. Pages 1-19 (up to and including Equation 6)

24.06.21

  1. *Brouwer, H., Delogu, F, Venhuizen, N. J., and Crocker, M. W. (2021). Neurobehavioral Correlates of Surprisal in Language Comprehension: A Neurocomputational Model. Frontiers in Psychology 12:110