Chomsky: "Three Models for the Description of Language" (1956)

This is my favorite Chomsky text, perhaps after Syntactic Structures. It contains a comparison of finite-state, phrase-structure, and context-sensitive languages; it also suggests that a transformational theory is the most illuminating generative story for English sentences.

A structural ambiguity; p. 118.

Garden Varieties

Among other things, the paper contains the following examples of formal languages (p. 115):

• "Mirror-image" sentences: aa, bb, abba, baab, aabbaa, …
• Echo sentences: aa, bb, abab, baba, aabaab, …
• Counting sentences: ab, aabb, aaabbb, …

The counting language is also used to show that the set of phrase-structure languages is a proper subset of the set of context-sensitive languages (p. 119).

A Markov model hidden states (and thus arbitrary dependence lengths); p. 116.

Irrelevant to Grammar

The paper also contains the familiar jump from a rejection of Markov models to a rejection of statistical models at large:

Whatever the other interest of statistical approximation in this sense may be, it is clear that it can shed no light on the problems of grammar. There is no general relation between the frequency of a string (or its component parts) and its grammaticalness. We can see this most clearly by considering such strings as

(14) colorless green ideas sleep furiously

which is a grammatical sentence, even though it is fair to assume that no pair of its words may ever have occurred together in the past. (p. 116)

Thus,

there is no significant correlation between order of approximation and grammaticalness. If we order the strings of a given length in terms of order of approximation to English, we shall find both grammatical and ungrammatical strings scattered throughout the list, from top to bottom. Hence the notion of statistical approximation appears to be irrelevant to grammar. (p. 116)

I suppose "order of approximation" here means "probability" rather literally the "order of the Markov model" (otherwise the this assertion doesn't make much sense).

Stockwell: "The Transformational Model of Generative or Predictive Grammar" (1963)

This essay by Robert Stockwell is a pretty much a rehash of Chomsky's ideas from Syntactic Structures, and as such quite boringly predictable. (Stockwell also thanks Chomsky for critical comments in the acknowledgments.) However, it does touch on some interesting issues on the last couple of pages, so I'll give a couple of excerpts here.

The essay is a part of an anthology called Natural Language and the Computer, which (in addition to being quite wonderfully techno-camp) is one of the most boldly and beautifully typeset books I've seen in a while.

The title page, with gargantuan print and the editor's name in a spacy font.

Some aspects of its graphical design has an undeniable 1950s feel to it; others look like a kind of 1960s Space Odyssey futurism; and others again look like they were taken straight out of Blade Runner. It's all quite remarkable.

Title page of Part 1, with an unmistakably 1950s italic font and a "etched" figure 1.

And then of course there's the always charming prospect of reliving a lecture taking stock of the computer revolution anno 1960. One of the other books in the same series is even called – I kid you not – The Foundations of Future Eletronics. How can you not love a 1961 book with that title?

At any rate, the essay that I'm interested in plods through the familiar details of Chomskyan grammar, pushing in particular the hard sell of transformational grammar over "immediate constituent" grammar (i.e., phrase structure grammars without any postprocessing).

The last section of the essay is called "Grammaticalness and Choice" and gets a bit into the issue of how sentences are selected, and what goes on in the head of the alleged "ideal speaker" of Chomskyan linguistics. This part contains a number of interesting quotes, and I'll provide some generous extracts below.

"Forced to Operate Empirically"

The first notion taken up in the section is that of grammaticality itself, which it seems that he sees some problems making empirically precise:

Presumably the notion of "grammatical sentence" is characterized by the grammar itself, since in principle we formulate our rules in such a way as to generate only and always such sentences. It is a question of some interest whether there is a possibility of characterizing this notion independently of the grammar. It seems extremely unlikely that there is, and we will be forced to operate empirically with the machinery of the grammar, treating each sentence that it generates as a hypothesis to be tested for grammaticalness against the the reaction of native speakers. For each sentence rejected, we either revise the grammar to exclude the sentence (if we believe the rejection is on proper grounds–that is, not motivated by school grammar and the like), or we revise the grammar to generate the sentence in some special status (i.e., as only partially well formed). Each sentence accepted is, of course, a confirmation of the validity of the rules up to that point. (p. 43)

I suppose what Stockwell has in mind here is that there might in principle exists some kind of objective test of grammaticality which could relieve us of having to trust laypeople to know the difference between "ungrammatical" and "nonsensical." (If you'll allow me a bit of self-citation, I've written a short paper on the idea of having such a distinction.)

Today, linguists might fantasize about such a test taking the form of an fMRI scan; in the 1980s, they would have imagined it as an EEG; and in the 1950s, a polygraph reading. But in the absence of such a test, we are forced to use live and conscious people even though

Informant reaction is difficult to handle, because such reactions involve much more than merely the question of grammaticalness. (p. 43)

We thus only have indirect access to the alleged grammatical engine of the brain.

"The Rest Takes Care of Itself"

After briefly considering a couple of borderline grammatical cases, Stockwell continues:

One might consider the utilization of grammar by the speaker as follows. The essence of meaning is choice; every time an element is chosen in proceeding through the rules, that choice is either obligatory (in which case it was not really a choice at all, since there were no alternatives), or it is optional (in which case the choice represented simultaneously both the positive decision and the rejection of all alternatives–the meaning of the choice inheres in the [sic] constrastive value of the chosen element as compared with all the possible choices that were rejected). (p. 44)

Oddly enough, Stockwell's meditation on the actual role and implementation of Chomskyan grammar in a person's behavior brings him around to confirming not only de Saussure's picture of meaning, but also Shannon's. I wonder whether he is aware of the implications of this.

He then goes on to consider an example:

Thus these are the choices involved in a simple sentences such

 Did the boy leave.

NP + VP	Obligatory
D + N	Obligatory
D == the	Optional
N == boy	Optional
aux + VP1	Obligatory
aux == past	Optional
VP1 == Vi	Optional
Vi == leave	Optional
Tintrg	Optional
Inversion of Te	Obligatory
Empty carrier for past	Obligatory
Rising intonation	Obligatory

Of the twelve choices, half are obligatory–either initiating the derivation, or following out obligatory consequences of optional choices. The additional rules of the phonetic component are nearly all obligatory. To include these would increase the obligatory choices to about twice the number of optional choices. In fact it is quite probable that in real discourse even the element the is obligatory (that is, the choice of the versus a seems quite predictable in a larger context). This would leave us with only five meaning-carrying (optional) choices. Everything else that goes into making up the sentence is in a valid sense perfectly mechanical, perfectly automatic. It can be argued that a grammar must maximize the obligatory elements and cut the optional choice to the barest minimum in order to get any reasonable understanding of how the human brain is capable of following complex discourse at all. That is, the hearer's attention is focused on matching, with his own generating machinery, the sequence of optional choices; since he has identical obligatory machinery, the rest takes care of itself. In this way, the same grammatical model accounts for both encoding and decoding. We do not need separate and distinct analogs for both sending and receiving messages. (pp. 44–45)

Again, this is oddly similar to the kind of generative model one would employ in information theory, and the notion of having a sparse language to minimize cognitive effort here takes the place of error-correction. But presumably, the philosophical difference is whether we need a source model (of the "optional choices") or only a channel model (of the "perfectly mechanical, perfectly automatic" choices).

"From Which He Knowingly Deviates"

This reading of the differences is backed up by his elaboration:

Encoding and decoding does not imply that a speaker of hearer proceeds step by step in any literal sense through the choices characterized by the grammar in order to produce or understand sentences. The capacities characterized by the grammar are but one contributing factor of undetermined extent in the total performance of the user of language. The grammar enumerates only well-formed sentences and deviant sentences, which, recognized as ranging from slightly to extremely deviant by the language user, are interpreted somehow by comparison with well-formed ones. The grammar enumerates sentences at random; it does not select, as the user does, just that sentences appropriate to a context. The grammar clacks relentlessly through the possible choices; the user starts, restarts, jumps the grammatical traces, trails off. A generative grammar is not a speaker-writer analog. It is a logical analog of the regularities to which the language user conforms or from which he knowingly deviates. (p. 45)

I take it that this entails that grammars are essentially and necessarily logical in nature, since their purpose is to describe the set of available sentences of the language rather than to predict their occurrence. From such a perspective, a probabilistic context-free grammar would be something of an oxymoron.

A logical and a probabilistic conception of grammar.

 

"A Natural Extension of Scholarly Grammar"

Again in perfectly orthodox fashion, Stockwell finally tips his hat at the impossibility of formulating discovery procedures and makes a strange claim about the machine-unfriendly nature of transformation grammars:

Although the title of this book suggests the machine processing of natural-language data, it should not be assumed that the transformational model of the structure of language is in any way geared to machines of any kind, either historically or in current development. On the contrary, it is a natural extension of traditional scholarly grammar, an effort to make explicit the regularities to which speakers of a language conform, which has been the focus of grammatical studies for over 2,500 years. The effort to formulate discovery procedures for the systematic inference of grammatical structure is quite recent; few if any transformationalists believe such a goal has any possibility of success–at least, not until much more is known about the kinds of regularities which grammarians seek to discover and to formulate in explicit terms. (p. 45)

That seems a bit odd in the light of, e.g., Victor Yngve's recollection of how people were swayed by Chomsky's grammatical analyses because they "read like computer programs."

Kuperberg: "Neural mechanisms of language comprehension" (2006)

Suppose I put you in front of a computer screen that flashes a word every second:

• The … cheese … was … eaten … by …

While you're reading, I record the electrical activity off the scalp of your head with an EEG scanner. Possibly, I also ask you to do something with the sentence when you're done reading, like judge its plausibility, or answer a question about it.

Once you've gotten used to this task, I perform a manipulation: Without warning, I insert a weird or unexpected word:

• The … cheese … was … eaten … by …the … cat …

When this happens, you'll obviously have to work harder than usual to make sense of this unexpected stimulus. This means more brain activity, and more brain activity means more electrical charge.

Typical Responses: The N400 and the P600

There are, in particular, two specific ways that the electrical activity at the scalp of your head changes measurably when this happens: You may exhibit an excess of negative electrical energy about 400 milliseconds the unexpected word, or an excess of positive electrical about 600 milliseconds after.

These two events are called the N400 and the P600. They can occur together, separately, or not at all.

The N400 was first described in 1980 by Martha Kutas and Steven A. Hillyard. They explained it as a a kind of "second look" effect and found that it was provoked by "semantic incongruity" (p. 204).

The P600 was described in 1992 by Lee Osterhout and Phillip J. Holcomb. They were explicitly interested in teasing apart syntactic from semantic effects, and they found that the P600 appeared specifically after syntactic anomalies like The librarian trusted to buy the books.

This was great news for the Chomskyan theory of language: At last, solid evidence that semantics and syntax are independent. And what could be more convincing to a linguist than "brain stuff"?

Delineation Problems

But of course, the story is a bit more complicated than that. In a wonderful paper from 2006, Gina R. Kuperberg reviews the large and growing pool of experimental findings related to the N400 and the P600.

Her conclusion is that the two electrical responses are the trace of two different processes, "one that links incoming semantic information with existing information stored in semantic memory, and another that combines relationships between people, objects and actions to construct new meaning" (p. 45).

If we want to evaluate a synthesis like that, we need to keep to separate issues apart: First, can we predict when the two different waveforms will come up? And second, if we can predict this, by what cues?

This dichotomy reflects the familiar problem of, on one hand, assessing whether people have stable intuitions about grammaticality, and, on the other hand, trying to articulate those intuitions in an adequate grammatical formalism. We can get both of these tasks wrong, independently of each other.

So here's what I want to do: I'll just give you a huge list with examples, and then you'll get a sense of where the N400 shows up, and where the P600 shows up. If it looks as if there is a system to this, then we can throw some grammatical vocabulary at this system; but first we need to get a sense of what the system is.

A Bunch of Examples

This section contains all of the examples that Kuperberg cites in her review. They come from a wide range of different sources, so I can't appropriately cite every one of them. I'll just repeat her examples without attribution.

The N400 was originally described as a reaction to semantic anomalies. It shows up in contrastive pairs like the following:

• It was his first day at work (baseline)
• He spread the warm bread with socks (strong N400)

It shows up strongly when you read sentences that are completely unambiguous as to what they are saying, but just say something really weird:

• The honey is being murdered (strong N400)

It's also visible when words are semantically permissible, but less expected:

• He mailed the letter without a stamp (baseline)
• He mailed the letter without a thought (moderate N400)

In fact, the N400 also appears when a sentence expresses completely legitimate assertions which just happen to be inconsistent with experience:

• Dutch trains are white (strong N400; they are in fact yellow)

The P600, on the other hand, was originally described as sensitive to syntactic violations. This conclusion was based on contrasts like the following:

• The broker hoped to sell the stock (baseline)
• The broker persuaded to sell the stock (strong P600)

Similarly, we find contrasts such as these:

• The doctor believed the patient was lying (baseline)
• The doctor charged the patient was lying (strong P600)

This morphosyntactic account of the causes of the P600 is also consistent with the fact that it responds to grammatical incongruence and weird word orders:

• The spoiled child throw the toys on the floor (strong P600)
• The expensive very tulip (strong P600)
• Jennifer rode a gray huge elephant (P600; compare huge gray)

Somewhat strangely, though, the predictability of the word that carries the incongruence can affect how strong the P600 effect is:

• Sie bereist den Land … (strong P600; Land is expected, but should be das)
• Sie befährt den Land … (milder P600)

The P600 can also be provoked by sentences in which the subject and object seem to be swapped or replaced by a wrong word:

• Every morning at breakfast the eggs would eat … (P600)
• Every morning at breakfast the eggs would plant … (P600)

This contrasts with the N400 effect that is visible when the word is merely unexpected:

• Every morning at breakfast the boys would plant … (N400)

A similar example is the following:

• The hearty meal was devoured … (baseline)
• The hearty meal was devouring … (P600)
• The dusty tabletops were devouring … (N400)

Or, again:

• Tyler cancelled the subscription (baseline)
• Tyler cancelled the birthday (N400)
• Tyler cancelled the tongue (N400 + P600)

This also has the consequence that when a cat flees a mouse, or when a javelin throws an athlete, you see a strong P600 effect rather than a N400:

• De kat die voor de muizen vluchtte … (P600)
• De speer heeft de athleten geworpen … (P600)

However, if the javelin summarizes the athletes, you get both a P600 and a N400 effect:

• De speer heeft de athleten opgesomd … (P600)

So it seems that when there is some sort of normal relationship between verb and object, but the sentence expresses the wrong one, both effects occur simultaneously:

• The trees that in the park played … (P600 + N400)
• The apple that in the tree climbed … (P600 + N400)

This doesn't depend on the distinction between subject and object, as can be seen by using a passive alternation:

• To make good documentaries cameras must interview … (P600)
• To make good documentaries cameras must be interviewed … (P600)

Another example of this contrast comes up if you let an elephant do various things to a tree: Topple it, prune it, or spoil it like a child:

• … dat de olifanten de bomen omduwden … (baseline)
• … dat de olifanten de bomen snoeiden … (P600)
• … dat de olifanten de bomen verwenden … (N400)

When a detective "stains" a banker instead of interrogating him, both effects also occur:

• … dass der Kommissar den Banker abhörte … (baseline)
• … dass der Kommissar den Banker abbeizte … (N400 + P600)

Also, when the verb understood gets an inanimate object as its agent, a P600 effect is visible:

• At long last, the man's pain was understood by the doctor (baseline)
• At long last, the man's pain was understood by the hypochondriac (weak N400)
• At long last, the man's pain was understood by the violinist (N400)
• At long last, the man's pain was understood by the medicine (strong N400 + P600)
• At long last, the man's pain was understood by the pens (strong N400 + P600)

A very nice example that also brings out the nature of the P600 waveform is the following:

• The novelist that the movie inspired … (P600)

This sentence is, strictly speaking, perfectly grammatical: A movie can inspire a novelist. However, it's much more probable to hear someone talk about a novel that inspired a movie, and something thus seems to have gone wrong with the sentence.

Interestingly, context can also heavily influence whether something counts as bizarre or as scrambled:

• [In a story about traveling] … the woman told the suitcase … (P600)
• [In a story about something else] … the woman told the suitcase … (N400)

Misspellings also seem to trigger a P600 effect if the context offers an obvious candidate for the correct word:

• In that library the pupils borrow bouks … (P600)
• The pillows were stuffed with bouks … (no P600)

I don't whether such examples also cause a N400 effect, but presumably, they do.

Sense-Making and Decoding

As may be apparent from the way I presented these examples, I'm not quite comfortable with the way that both the traditional accounts and Kuperberg's paper talks about "what the brain does" and "what the P600 picks up on." I think that we can largely make sense of the N400 and the P600 waveforms in terms of what kind of repair practices the sentences suggest.

More precisely, you can react to an unexpected sentence in two ways: Either, you can suspect that it was scrambled or otherwise corrupted due to noise, or you can believe that it came over uncorrupted, but just expresses a really weird idea. You might be pushed into the first hypothesis if there is a really obvious nearby expression that makes much more sense, and into the second if there isn't.

Looking at it this way quite accurately explains the differences between the two effects, I think, and essentially doesn't invoke any grammatical notions. Instead, we can think of comprehension as a kind of Bayesian decoding process along the following lines:

1. Recover the intended message from the received codeword.
2. Find a reasonable interpretation of the intended message.

When the received codeword corresponds unproblematically to a message, the first step is fast, and we can proceed directly to the interpretation within the first 500 milliseconds.

If you then afterwards find that the intended message is really weird, you can either go back and check whether there really wasn't corrupted, or you can just work harder trying to interpret it. The first response will yield a P600 effect, and the second an N400.

An advantage of this story is also that it accounts for the strange fact that "semantics" should be processed before "syntax." The order of the N400 and the P600 may be due to the fact that reconstructing a likely message (like so many backwards-reasoning tasks), is much more computationally expensive interpreting one. Kuperberg also hints at this possibility by attributing the P600 to "combinatory processing."

Of course, I also like this story because it doesn't drive a wedge in between syntax and semantic when there doesn't have to be one. But you can disagree with me on that.

Robin Lakoff's introduction (2004) to Language and Woman's Place (1975)

In 2004, a new annotated edition of Robin Lakoff's Language and Woman's Place came out. I addtion to the orignal text, this new edition includes a number of essays on the topics raised in the book as well as a new introduction by Lakoff herself.

The Frustration of a Comskyan

The introduction is interesting for several reasons. One of them is that it contains an interesting personal account of the ideas that were splitting transformational grammar apart in the era of Aspects:

By the late 1960s it had become clear to several of us that Chomsky's linguistic revolution wasn't the recolution in which we had enlisted. Chomsky had promised us a theory that would make language a "window into the mind." But within standard transformational theory that possibility could be realized only to a very limited dregree, if at all. While investigators could use their minds as interpretive instruments—to judge the grammaticality or semantic similarity of sentences—they were not permitted to investigate meaning, much less a speaker's intention in uttering a sentence in a particular form, or the effect of that utterance on the addressee.

Consequently, Lakoff and others got the idea of starting from meaning representations of a certain form and then deriving syntactic form out of that. To the extent that it ever was a single or coherent theory, this new framework is what we now call "generative semantics":

We devised rules and representations to relate externally accessible linguistic forms to mental states—for example, desires, assumptions, and personal identities—while retaining the Chomskyan belief in the primacy of the syntactic component of the grammar. Deep structure got deeper, wider, and more complex.

"Deep structure got deeper"—what a wonderful summary. Remember also George Lakoff's comment that he and his gang just wanted to be "good little comskyans," quoted in The Linguistics Wars. How wrong they both were—about what they were doing, and how much linguistics would change.

Grammar Unlimited?

Interestingly, the orthodox Chomskyan argument against this augmented view of grammar was, in Lakoff's rendering:

If you followed generative semantics to its logical conclusion, everything speakers know about the world would have to be included within the transformational component, which therefore would become infinite. [...]

Not necessarily, said the generative semanticists. The linguistic grammar need only include those aspects of the extralinguistic world that have direct bearing on grammatical form: just a small subset of everything. [...] But we still had to answer, at least to our own satisfaction, the question that these claims raised: What parts of our psychological and social reality did require linguistic encoding, in at least some languages?

Lakoff's point in the essay is of course that gender is one of the important variables of linguistic expression—not just "in a few 'exotic' languages (Japanese, Dyirbal, Arawak, and Koasati)," but in solid, run-of-the-mill English as well.

But at the same time, the quote points right at the big taboo of linguistics, whether or not Lakoff herself intends it to do so: Once we admit that syntax can't be isolated from meaning, the floodgates are open to seeing that there really isn't any such thing as a "language" at all; the difference between syntax and anthropology is really more about differences in interests than about anything inherent in the "object of study."

Is Linguistics Linguistics?

Lakoff is also aware that something in transformational grammar and generative semantics was holding it back from saying anything intelligent about discourse structure on a larger level:

Linguists spoke on occasion of "structure above (or beyond) the sentence level," but mostly about how it couldn't be done. When we attempted it, we thought of larger units as concatenations of sentences: S+S+S ..., rather than as structures with rules of their own, wholes different from the sum of their parts.

While I think that this has something to do with the game of showing-and-hiding that linguistics necessarily entails, Lakoff seems to attribute it more to the fact that different tools fit different situations:

While a generation ago, "structure above the sentence level" had the status of the basilisk (mythical and toxic), now it is an accepted area of linguistics [...] These analyses made it clear that discourse should be understood not as concatenations of S's, but as language directed toward particular interactive and psychological purposes.

So now we're OK, the message seems to be; we just had to realize that a different set of concepts was needed (turntaking, politeness, power, identity, etc.).

While I agree that conversational analysis has something new and interesting to say about language use, I also think that there's something genuinely wrong about saying that it does the same thing as Chomskyan grammar, only with a different tool. It's not just a shift of attention or of measuring equipment, it's a shift of standards, mindset, ethics, and goals.

Not that there's anything wrong with either hermeneutics or with mathematics—it's just that they are never going to be unified into a single methodology. There is a tension between the picture of what counts as data and valid argument that Lakoff's book drew attention to, and I don't think it's a tension that can or should be resolved.