Newman et al.: "An Event-Related fMRI Study of Syntactic and Semantic Violations" (2001)

This paper reports on a brain imaging study in which people were given either ordinary English sentences or sentences with one of two types of error (p. 349):

• Yesterday I sailed Todd's hotel to China. (semantic violation)
• Yesterday I cut Max's with apple. (syntactic violation)

The sentences were presented one word at a time. They don't seem to say when recordings started, but they do mention "critical" words (p. 139).

The results were the following:

The syntactic errors lit up Brodmann's area 6, which includes the premotor cortex. It also activated the a spot in superior temporal gyrus which was either Brodmann's area 21 or 22.

The semantic errors activated a number of regions, including several quite frontal regions. The strongest activation was here inside the the fissure between the two halves of the brain, that is, in the medial section of the cortex.

Here a table:

And here is a picture (Fig. 1, p. 352):

The picture is ugly as hell, but I like that they are so infatuated with this beautiful brain: "The structural image used is from one subject, chosen for its particularly high quality … the choice of one anatomical image over another does not alter the localization of the results—the differences are purely aesthetic." (From the caption of Fig. 1.)

Regarding the syntactic observations, they comment:

Our findings suggest that the superior frontal gyri, including not only premotor cortex, but those portions of it which likely correspond to the supplementary motor area, are involved in the processing of syntactic phrase structure violations. While this is an unusual finding, … it is not unique: Ni et al. (2000, Experiment 2) found superior frontal gyrus activation in response to morphosyntactic violations. (p. 355)

What do we make of this? Well, it would seem that what you do when you read I cut with cake knife is quite literally comparable to pulling an emergency brake: You use the same inhibitory resources as those that are involved in holding yourself back from pushing the wrong button in a stop-go experiment, or training yourself to perform a complex series of movements with your fingers.

Chomsky: "Three Models for the Description of Language" (1956)

This is my favorite Chomsky text, perhaps after Syntactic Structures. It contains a comparison of finite-state, phrase-structure, and context-sensitive languages; it also suggests that a transformational theory is the most illuminating generative story for English sentences.

A structural ambiguity; p. 118.

Garden Varieties

Among other things, the paper contains the following examples of formal languages (p. 115):

• "Mirror-image" sentences: aa, bb, abba, baab, aabbaa, …
• Echo sentences: aa, bb, abab, baba, aabaab, …
• Counting sentences: ab, aabb, aaabbb, …

The counting language is also used to show that the set of phrase-structure languages is a proper subset of the set of context-sensitive languages (p. 119).

A Markov model hidden states (and thus arbitrary dependence lengths); p. 116.

Irrelevant to Grammar

The paper also contains the familiar jump from a rejection of Markov models to a rejection of statistical models at large:

Whatever the other interest of statistical approximation in this sense may be, it is clear that it can shed no light on the problems of grammar. There is no general relation between the frequency of a string (or its component parts) and its grammaticalness. We can see this most clearly by considering such strings as

(14) colorless green ideas sleep furiously

which is a grammatical sentence, even though it is fair to assume that no pair of its words may ever have occurred together in the past. (p. 116)

Thus,

there is no significant correlation between order of approximation and grammaticalness. If we order the strings of a given length in terms of order of approximation to English, we shall find both grammatical and ungrammatical strings scattered throughout the list, from top to bottom. Hence the notion of statistical approximation appears to be irrelevant to grammar. (p. 116)

I suppose "order of approximation" here means "probability" rather literally the "order of the Markov model" (otherwise the this assertion doesn't make much sense).

Drum: "Change, Meaning, and Information" (1957)

This 1957 article is one of those countless papers by people in linguistics trying helplessly to say something non-trivial about the relationship between information theory and linguistics. The central claim of the paper is that there are several kinds of predictability or redundancy at play in language, and that "meaning" should be seen as one of them.

More precisely, certain sentences are predictable on account of their content, so that

the "meaning"—the "what we know about it"–in a message has some direct effect upon the amount of information transmitted. (p. 166)

However, this notion is dangerously confused and conflated with the idea that more predictability equals more meaning at several points in the paper.

At any rate, Dale places semantics among several "levels" of redundancy, including a historically interesting division between "formal" and "contextual" syntax:

In the sentence, "The chair is __________," many words could be correctly used in terms of grammar; but in terms of the context of the sentence, only a very few apply. "The chair is clock" is correct grammatically, though it is nonsense in even the broadest contexts. (p. 167)

Further,

A person knowing English grammar but not word meanings, might very well write "The chair is clock," but would never do so if he knew the meanings involved. (The relationship of syntactic to contextual redundancy is much that of validity in logic to truth, incidentally). (p. 168)

A strange counterfactual, but quite revealing.

Kuperberg: "Neural mechanisms of language comprehension" (2006)

Suppose I put you in front of a computer screen that flashes a word every second:

• The … cheese … was … eaten … by …

While you're reading, I record the electrical activity off the scalp of your head with an EEG scanner. Possibly, I also ask you to do something with the sentence when you're done reading, like judge its plausibility, or answer a question about it.

Once you've gotten used to this task, I perform a manipulation: Without warning, I insert a weird or unexpected word:

• The … cheese … was … eaten … by …the … cat …

When this happens, you'll obviously have to work harder than usual to make sense of this unexpected stimulus. This means more brain activity, and more brain activity means more electrical charge.

Typical Responses: The N400 and the P600

There are, in particular, two specific ways that the electrical activity at the scalp of your head changes measurably when this happens: You may exhibit an excess of negative electrical energy about 400 milliseconds the unexpected word, or an excess of positive electrical about 600 milliseconds after.

These two events are called the N400 and the P600. They can occur together, separately, or not at all.

The N400 was first described in 1980 by Martha Kutas and Steven A. Hillyard. They explained it as a a kind of "second look" effect and found that it was provoked by "semantic incongruity" (p. 204).

The P600 was described in 1992 by Lee Osterhout and Phillip J. Holcomb. They were explicitly interested in teasing apart syntactic from semantic effects, and they found that the P600 appeared specifically after syntactic anomalies like The librarian trusted to buy the books.

This was great news for the Chomskyan theory of language: At last, solid evidence that semantics and syntax are independent. And what could be more convincing to a linguist than "brain stuff"?

Delineation Problems

But of course, the story is a bit more complicated than that. In a wonderful paper from 2006, Gina R. Kuperberg reviews the large and growing pool of experimental findings related to the N400 and the P600.

Her conclusion is that the two electrical responses are the trace of two different processes, "one that links incoming semantic information with existing information stored in semantic memory, and another that combines relationships between people, objects and actions to construct new meaning" (p. 45).

If we want to evaluate a synthesis like that, we need to keep to separate issues apart: First, can we predict when the two different waveforms will come up? And second, if we can predict this, by what cues?

This dichotomy reflects the familiar problem of, on one hand, assessing whether people have stable intuitions about grammaticality, and, on the other hand, trying to articulate those intuitions in an adequate grammatical formalism. We can get both of these tasks wrong, independently of each other.

So here's what I want to do: I'll just give you a huge list with examples, and then you'll get a sense of where the N400 shows up, and where the P600 shows up. If it looks as if there is a system to this, then we can throw some grammatical vocabulary at this system; but first we need to get a sense of what the system is.

A Bunch of Examples

This section contains all of the examples that Kuperberg cites in her review. They come from a wide range of different sources, so I can't appropriately cite every one of them. I'll just repeat her examples without attribution.

The N400 was originally described as a reaction to semantic anomalies. It shows up in contrastive pairs like the following:

• It was his first day at work (baseline)
• He spread the warm bread with socks (strong N400)

It shows up strongly when you read sentences that are completely unambiguous as to what they are saying, but just say something really weird:

• The honey is being murdered (strong N400)

It's also visible when words are semantically permissible, but less expected:

• He mailed the letter without a stamp (baseline)
• He mailed the letter without a thought (moderate N400)

In fact, the N400 also appears when a sentence expresses completely legitimate assertions which just happen to be inconsistent with experience:

• Dutch trains are white (strong N400; they are in fact yellow)

The P600, on the other hand, was originally described as sensitive to syntactic violations. This conclusion was based on contrasts like the following:

• The broker hoped to sell the stock (baseline)
• The broker persuaded to sell the stock (strong P600)

Similarly, we find contrasts such as these:

• The doctor believed the patient was lying (baseline)
• The doctor charged the patient was lying (strong P600)

This morphosyntactic account of the causes of the P600 is also consistent with the fact that it responds to grammatical incongruence and weird word orders:

• The spoiled child throw the toys on the floor (strong P600)
• The expensive very tulip (strong P600)
• Jennifer rode a gray huge elephant (P600; compare huge gray)

Somewhat strangely, though, the predictability of the word that carries the incongruence can affect how strong the P600 effect is:

• Sie bereist den Land … (strong P600; Land is expected, but should be das)
• Sie befährt den Land … (milder P600)

The P600 can also be provoked by sentences in which the subject and object seem to be swapped or replaced by a wrong word:

• Every morning at breakfast the eggs would eat … (P600)
• Every morning at breakfast the eggs would plant … (P600)

This contrasts with the N400 effect that is visible when the word is merely unexpected:

• Every morning at breakfast the boys would plant … (N400)

A similar example is the following:

• The hearty meal was devoured … (baseline)
• The hearty meal was devouring … (P600)
• The dusty tabletops were devouring … (N400)

Or, again:

• Tyler cancelled the subscription (baseline)
• Tyler cancelled the birthday (N400)
• Tyler cancelled the tongue (N400 + P600)

This also has the consequence that when a cat flees a mouse, or when a javelin throws an athlete, you see a strong P600 effect rather than a N400:

• De kat die voor de muizen vluchtte … (P600)
• De speer heeft de athleten geworpen … (P600)

However, if the javelin summarizes the athletes, you get both a P600 and a N400 effect:

• De speer heeft de athleten opgesomd … (P600)

So it seems that when there is some sort of normal relationship between verb and object, but the sentence expresses the wrong one, both effects occur simultaneously:

• The trees that in the park played … (P600 + N400)
• The apple that in the tree climbed … (P600 + N400)

This doesn't depend on the distinction between subject and object, as can be seen by using a passive alternation:

• To make good documentaries cameras must interview … (P600)
• To make good documentaries cameras must be interviewed … (P600)

Another example of this contrast comes up if you let an elephant do various things to a tree: Topple it, prune it, or spoil it like a child:

• … dat de olifanten de bomen omduwden … (baseline)
• … dat de olifanten de bomen snoeiden … (P600)
• … dat de olifanten de bomen verwenden … (N400)

When a detective "stains" a banker instead of interrogating him, both effects also occur:

• … dass der Kommissar den Banker abhörte … (baseline)
• … dass der Kommissar den Banker abbeizte … (N400 + P600)

Also, when the verb understood gets an inanimate object as its agent, a P600 effect is visible:

• At long last, the man's pain was understood by the doctor (baseline)
• At long last, the man's pain was understood by the hypochondriac (weak N400)
• At long last, the man's pain was understood by the violinist (N400)
• At long last, the man's pain was understood by the medicine (strong N400 + P600)
• At long last, the man's pain was understood by the pens (strong N400 + P600)

A very nice example that also brings out the nature of the P600 waveform is the following:

• The novelist that the movie inspired … (P600)

This sentence is, strictly speaking, perfectly grammatical: A movie can inspire a novelist. However, it's much more probable to hear someone talk about a novel that inspired a movie, and something thus seems to have gone wrong with the sentence.

Interestingly, context can also heavily influence whether something counts as bizarre or as scrambled:

• [In a story about traveling] … the woman told the suitcase … (P600)
• [In a story about something else] … the woman told the suitcase … (N400)

Misspellings also seem to trigger a P600 effect if the context offers an obvious candidate for the correct word:

• In that library the pupils borrow bouks … (P600)
• The pillows were stuffed with bouks … (no P600)

I don't whether such examples also cause a N400 effect, but presumably, they do.

Sense-Making and Decoding

As may be apparent from the way I presented these examples, I'm not quite comfortable with the way that both the traditional accounts and Kuperberg's paper talks about "what the brain does" and "what the P600 picks up on." I think that we can largely make sense of the N400 and the P600 waveforms in terms of what kind of repair practices the sentences suggest.

More precisely, you can react to an unexpected sentence in two ways: Either, you can suspect that it was scrambled or otherwise corrupted due to noise, or you can believe that it came over uncorrupted, but just expresses a really weird idea. You might be pushed into the first hypothesis if there is a really obvious nearby expression that makes much more sense, and into the second if there isn't.

Looking at it this way quite accurately explains the differences between the two effects, I think, and essentially doesn't invoke any grammatical notions. Instead, we can think of comprehension as a kind of Bayesian decoding process along the following lines:

1. Recover the intended message from the received codeword.
2. Find a reasonable interpretation of the intended message.

When the received codeword corresponds unproblematically to a message, the first step is fast, and we can proceed directly to the interpretation within the first 500 milliseconds.

If you then afterwards find that the intended message is really weird, you can either go back and check whether there really wasn't corrupted, or you can just work harder trying to interpret it. The first response will yield a P600 effect, and the second an N400.

An advantage of this story is also that it accounts for the strange fact that "semantics" should be processed before "syntax." The order of the N400 and the P600 may be due to the fact that reconstructing a likely message (like so many backwards-reasoning tasks), is much more computationally expensive interpreting one. Kuperberg also hints at this possibility by attributing the P600 to "combinatory processing."

Of course, I also like this story because it doesn't drive a wedge in between syntax and semantic when there doesn't have to be one. But you can disagree with me on that.