“Prediction during language comprehension: what is next?”
Highlights:
- During language processing, comprehenders predict upcoming linguistic input. These predictions draw on many sources of information including the preceding sentence context.
- We review a substantial body of evidence which has explored the information used and generated in the computation of prediction, as well as the constraints.
- Further progress in understanding prediction in language comprehension will likely require developing mechanistic explanations. We discuss four research questions which may help to guide the development of these explanations.
Prediction is often regarded as an integral aspect of incremental language comprehension, but little is known about the cognitive architectures and mechanisms that support it.
We review studies showing that listeners and readers use all manner of contextual information to generate multifaceted predictions about upcoming input.
The nature of these predictions may vary between individuals owing to differences in language experience, among other factors.
We then turn to unresolved questions which may guide the search for the underlying mechanisms.
(i) Is prediction essential to language processing or an optional strategy?
(ii) Are predictions generated from within the language system or by domain-general processes?
(iii) What is the relationship between prediction and memory?
(iv) Does prediction in comprehension require simulation via the production system?
We discuss promising directions for making progress in answering these questions and for developing a mechanistic understanding of prediction in language.
M nodes indicate neural activity encoding a model of the level below. These models generate top-down (TD) predictions (Pred., blue arrows) about the level below. C nodes indicate comparisons between the activity predicted by the M of the level above and the actual activity at the current level. These comparisons result in top-down (TD) errors (red arrows pointing down) which adjust the estimate of activity at a given level and, crucially, bottom-up (BU) prediction errors (red errors pointing up) which serve to improve model predictions at the next level up.
Nodes at higher levels receive inputs from multiple instances of prediction errors at lower levels.
Predictive coding
The idea that human minds and brains predict and learn from error has a long history in cognitive science. Inspired by information-theoretic approaches to efficient message/video transmission, Barlow argued that perception encodes sensory stimuli in a way that reduces redundancy and transmits those signals which are least redundant, in other words those that are least predictable. These ideas were the direct precursors of the modern-day predictive coding account.
The term ‘predictive coding‘ is sometimes used interchangeably with ‘prediction’ or ‘predictive processing’ to refer to any instance of perception or cognition where a response is reduced for input that is likely given ‘top-down’ information (e.g., the statistics of the environment) than input that is unlikely. […] The key features of this proposal are listed below.
(i) Hierarchical organization: sensory inputs (e.g., sound) are represented at the lowest levels, whereas more abstract information (e.g., word meanings) is represented at higher levels of the hierarchy (in simulations, these levels correspond to layers of a neural network model, but conceptually these levels could map onto different brain regions and/or different cortical layers within a region).
(ii) Top-down predictions are transmitted from higher levels to lower levels of the hierarchy: each level predicts the responses in the level immediately below via feedback connections.
(iii) Bottom-up prediction errors travel from lower levels to higher levels of the hierarchy: each level transmits the discrepancy between the predicted response and the estimated actual response to the level immediately above it via feedforward connections.
(iv) Prediction errors are used to update the response at each level and generate the next prediction, thus allowing the mind to process/infer the current input and continuously adapt to its environment.
(v) Predictions and errors are carried by distinct populations of functional units: the mapping to neuroanatomy is purely speculative at this point, but some functional distinction will be necessary to allow both forward cascades of errors and backward flow of predictions.
(vi) Bayesian inference and precision-weighting: the predictive coding algorithm approximates Bayesian inference. The outcome of predictive computations corresponds to probability distributions over predictions, as opposed to singular predictions, potentially in the form of probabilistic population codes. Following Bayesian principles, the posterior probabilities of the internal model, at each level, capture the uncertainty of both the bottom-up input and the top-down predictions. When the sensory input is noisy or prior knowledge is sparse, they are said to have low precision (defined as inverse variance) and are down-weighted in the update computation.
This proposal finds support in neuroanatomy, computational simulations which match response patterns of recordings directly from animal cortical cells, and a plethora of findings that human neural response patterns are consistent with predictive coding.
Furthermore, recent computational work shows that a recurrent neural network trained to optimize energy efficiency, within a predictive environment, self-organizes into distinct subsets of neurons, one carrying errors and the other carrying predictions. However, many of the empirical studies that are attributed to predictive coding (especially in higher-level cognitive domains) are also consistent with other possible models of predictive processing….
Schematic summary of inputs to, outputs of, and constraints on the computation of prediction during language comprehension. An asterisk (*) indicates areas of ongoing debate/investigation.
(A) Essential versus optional.
Blue bars schematically represent the probabilities of upcoming words (w1 = word 1, w2 = word 2, etc.). Only four words are shown, but the probabilities are distributed over all possible continuations and encompass meaning, form, etc. On an account of prediction as an essential component of language comprehension, predictions have higher entropy when the context is uninformative and/or their cognitive resources are taxed (e.g., unnaturally fast presentation, dual task). On an account of prediction as an optional strategy, no probabilities are computed under those circumstances.
(B) Language-specific versus domain-general.
Blue circles represent language network processes. Red circles represent multiple-demand network processes. Some aspects of linguistic prediction are likely implemented in local circuits (blue circles and arrows reflect a predictive coding-like architecture in which predictions travel up and errors travel down the hierarchy). Domain-general cognitive control may play a role under specific circumstances (e.g., when there are errors in the input or when flexible switching between contexts is required), and the locus of its influence is unknown (as indicated by dashed arrows connected to either errors or predictions).
(C) Prediction and memory.
The three stages of the cycle represent (clockwise)
(i) the generation of probabilistic predictions in response to language input (blue bars represent probabilities),
(ii) the change in memory representations that results from the prediction that was generated and its relationship to the received input (e.g., a highly predicted input that is disconfirmed by the received input is still ‘boosted’ in memory), and
(iii) the lossy memory for the context used to predict a subsequent input, which is determined by the state of an individual’s memory for language.
(D) Prediction as production.
Comprehension and the prediction processes that support it receive sensory/perceptual inputs and extract meaning. Production consists of transforming the intended meaning of a speaker into motor actions. Although they tap into the same meaning representations, the low-level input/output representations of production (i.e., motor movements of speech articulation, writing, signing) and comprehension (vision, audition, touch) are distinct. Whether prediction during comprehension involves forward-simulating production, and how far down the production pipeline this simulation might go, is an open question.
Abbreviations: EF, executive function; MD, multiple-demand system.
