synopsis
This course will examine neurocomputational (or connectionist) models of human language processing. We will start from biological neurons, and show how their processing behaviour can be modelled mathematically. The resulting artificial neurons will then be wired together to form artificial neural networks, and we will discuss how such networks can be applied to build neurocomputational models of language learning and language processing. It will be shown that such models effectively all share the same computational principles, and that any differences in their behaviour is driven by differences in the representations that they process and construct. Near the end of the course, we will use the accumulated knowledge to construct a psychologically plausible neurocomputational model of incremental (word-by-word) language comprehension that constructs a rich utterance representation beyond a simple syntactic derivation or semantic formula.
course overview
Connectionist Language Processing is a course taught in the Department of Language Science and Technology at Saarland University. It is open for master-level students.
Lecturer: Harm Brouwer
<me-at-hbrouwer.eu>
TA: Christoph Aurnhammer
<aurnhammer-at-coli.uni-saarland.de>
Time: Tuesday 14:15-15:45;
Thursday 14:15-15:45
Place: Online (Microsoft Teams)
Start: 20.04.21
Exam: Tuesday, July 20,
14:00-16.00
Credits: 6 CP
Resit Exam: Wednesday,
October 13, 10:00-12.00
Credits: 6 CP
Registration: Send me
an email to enrol for the course
Format and Requirements:
- The course will be taught online with a strong focus on self-study;
- Slides and suggested reading materials will be made available each week, and lecture sessions will serve to discuss these in an interactive Q&A style manner;
- Tutorials will be made available after each lecture, and you should try to complete them as far as possible before the corresponding tutorial session, where we will interactively discuss them in a Q&A style manner;
- Completed tutorial sheets should be handed in before the next lecture;
- You are expected to attend all lecture and tutorial sessions, and to complete all tutorial sheets.
schedule
This is the provisional course schedule. See below for suggested background literature.
| Date | Topic |
|---|---|
|
|
|
| 20.04.21 | Lecture 1: Introduction to Connectionism and the Brain |
| 22.04.21 | Tutorial 1: Introduction to Neural Networks in MESH |
|
|
|
| 27.04.21 | Lecture 2: A Primer on Linear Algebra |
| 29.04.21 | Lecture 3: Learning in Single-layer Networks |
|
|
|
| 04.05.21 | Lecture 4: Training Multi-layer Networks |
| 06.05.21 | Tutorial 2: Training Multi-layer Networks |
|
|
|
| 11.05.21 | Lecture 5: Reading Aloud |
| 13.05.21 | (no class: Christi Himmelfahrt) |
|
|
|
| 18.05.21 | Tutorial 3: Reading Aloud |
| 20.05.21 | Lecture 6: English Past Tense |
|
|
|
| 25.05.21 | Tutorial 4: English Past Tense |
| 27.05.21 | Lecture 7: Simple Recurrent Networks I |
|
|
|
| 01.06.21 | Lecture 8: Simple Recurrent Networks II |
| 03.06.21 | (no class: Fronleichnam) |
|
|
|
| 08.06.21 | Lecture 9: Recurrent Neural Networks as Models of Sentence Processing (Christoph Aurnhammer) |
| 10.06.21 | Tutorial 5: Simple Recurrent Networks |
|
|
|
| 15.06.21 | Lecure 10: Modeling the Electrophysiology of Language Comprehension |
| 17.06.21 | Lecture 11: Situation Modeling using Microworlds |
|
|
|
| 22.06.21 | Tutorial 6: Expectation-based Comprehension I |
| 24.06.21 | (no class) |
|
|
|
| 29.06.21 | Lecture 12: Modeling the Neurobehavioral Correlates of Comprehension-centric Surprisal |
| 01.07.21 | Tutorial 7: Expectation-based Comprehension II |
|
|
|
| 06.07.21 | Lecture 13: Course Summary |
| 08.07.21 | (no class) |
|
|
|
| 13.07.21 | Q&A |
| 15.07.21 | (no class) |
|
|
|
| 20.07.21 | Exam |
suggested literature
20.04.21
- Plunkett K, and Elman J. (1997). Exercises in rethinking innateness: A Handbook for Connectionist Simulations. Cambridge, MA: MIT Press. Chapter 2.
27.04.21
- Jordan, M. I. (1986). An introduction to linear algebra in parallel distributed processing. In: Rumelhart, D. E., and McClelland, J. L., et al. Parallel Distributed Processing, Vol. 1, pp. 365-422.
29.04.21
- Plunkett K, and Elman J. (1997). Exercises in rethinking innateness: A Handbook for Connectionist Simulations. Cambridge, MA: MIT Press. Chapter 1.
- *Brouwer, H. (2014). The Electrophysiology of Language Comprehension: A Neurocomputational Model. PhD thesis. University of Groningen, Groningen, The Netherlands. Appendix A (up to and including A.2.2)
04.05.21
- Plunkett K, and Elman J. (1997). Exercises in rethinking innateness: A Handbook for Connectionist Simulations. Cambridge, MA: MIT Press. Chapters 1 and 4.
- *Brouwer, H. (2014). The Electrophysiology of Language Comprehension: A Neurocomputational Model. PhD thesis. University of Groningen, Groningen, The Netherlands. Appendix A (up to and including A.2.3)
11.05.21
- Plaut, D. C., McClelland, J. L., Seidenberg, M. S., and Patterson, K. (1996). Understanding normal and impaired word reading: Computational principles in quasi-regular domains. Psychological Review, 103, pp. 56-115. Pages 1-19 (up to and including Summary)
20.05.21
- Plunkett K, and Marchman V. A. (1991). U-shaped learning and frequency effects in a multi-layered perceptron: Implications for child language acquisition. Cognition, 38(1). pp. 43-102. Pages 1-15 (up to and including 3.1)
27.05.21
- Elman, J. (1990). Finding structure in time. Cognitive Science, 14, pp. 179-211.
- *Brouwer, H. (2014). The Electrophysiology of Language Comprehension: A Neurocomputational Model. PhD thesis. University of Groningen, Groningen, The Netherlands. Appendix A (up to and including A.2.4)
01.06.21
- Elman, J. (1991). Distributed representations, simple recurrent networks, and grammatical structure. Machine Learning, 7, pp. 195-225.
- Elman, J. (1993). Learning and development in neural networks: the importance of starting small Cognition, 48, pp. 71-99.
15.06.21
- *Brouwer, H., Crocker, M. W., Venhuizen, N. J., and Hoeks, J. C. J. (2017). A Neurocomputational Model of the N400 and the P600 in Language Processing. Cognitive Science, 41(S6), pp. 1318-1352.
17.06.21
- *Venhuizen, N. J., Crocker, M. W., and Brouwer, H. (2019). Expectation-based Comprehension: Modeling the interaction of world knowledge and linguistic experience. Discourse Processes, 56:3, pp. 229-255. Pages 1-19 (up to and including Equation 6)
29.06.21
- *Brouwer, H., Delogu, F, Venhuizen, N. J., and Crocker, M. W. (2021). Neurobehavioral Correlates of Surprisal in Language Comprehension: A Neurocomputational Model. Frontiers in Psychology 12:110