I agree wholeheartedly with everything Asad said.
It's often mentioned that NLP turned away from "the old questions of symbolic representations" at the same time as there was relatively suddenly a huge increase in available electronic data (eg. the web) and in computing power to process it. But to be more precise, what we got a lot more of was electronic data about which words appear next to which other words, and linguists have long had a strong hunch that these aren't actually the data which most relevantly characterise a sentence.
It turned out that suddenly the results of the "old methods" (i.e. combining our best theory of which relationships really matter in a sentence, with the negligible data available about these relationships) were inferior to those of the "new methods" (i.e. combining a theory of which relationships really matter which was known to be not-quite-right, with the bucketloads of data available about those), and all the engineers flocked to these new methods, because they worked better. (Who would blame them?!) But I don't think this makes the theory that was known to be not-quite-right any better than we thought.
It might be argued that analysing the "which words are next to which other words" data as NLPers do is relevant for the language acquisition problem that linguists worry about, because this is the data that is available to human infants. I think this is a serious case of missing the point, however, because the infant comes equipped with "the right theory" (by definition) of which relationships really matter, so he/she is not in the same position as the engineers. (For subproblems at the phonological/morphological level, the situation might be more analogous, but I'm thinking more about sentence-level properties mainly.)
An interesting article on this topic, that Philip has mentioned a couple of times in class, is this one by Steven Abney:
http://www.ling.umd.edu/locker/ComputationalLearning/Abney95c.pdf
As may well be clear from the above, I happen to disagree with almost every interesting claim in there, but it does at least get you thinking. Some may remember rather lively discussions arising from this paper in last year's seminar.
Tuesday, September 23, 2008
What do you want?
Philip's last question in class ("Tell me what you want!" to paraphrase) has spurred this drive-by post. I think, to be completely blunt, while the goal of this course appears to reside in presenting NLP techniques to linguists, I'm not sure that this means of presenting it really serves that purpose. The field of linguistics doesn't seem to work that way: "present me with a set of hammers, and I will find nails." NLP applications have worked that way, because they are engineering. Linguistics, particularly of the sort practised at UMD, is not.
What linguists have done is that they've found the nails first, and if you are a critic, you'd say that they've tried to bang the nails in with their fists. (If you are not a general critic of UMDian linguistics, you'd say that nails analogy is bogus, and I might actually agree with you, but that's the subject of a different post and maybe a different course.) Be that as it may, to accomplish the goal of making this class relevant to linguists, it really has to be approached from the opposite direction of an NLP applications course.
That is to say, it has to be shown that these techniques are in fact the correct set of hammers, and that means starting from the nail. The linguist's accusation against attempts at importing NLP concepts has always been that the hammer is always too big or too little. To reiterate, you can't really say, "Well, I have these techniques, they use data!!! Surely you can find something." The linguist will wonder, as linguists often do, why s/he needs more than s/he already has, and whether LSA is just too big a hammer.
Let's take syntax. A question that seems to have been a biggie for the past little while here at UMD is the "Agree" issue: do we need a special syntactic operation like "Agree"? It's still not clear that NLP techniques serve to resolve these high-level questions (eg, what operations exist and are minimally necessary). And if they don't, then one has to find lower-level questions, perhaps the mechanics of the syntax-semantics interface. But that requires NLP techniques that can map strings to at least the First Order Predicate Calculus in order to do anything at the sophistication that semanticists use.
Since data driven techniques have either sacrificed this approach out of expediency or found them impossible to support, it's hard to make the case for the involvement of NLP in at least those two domains of linguistic investigation (syntax and semantics). I would suggest that the place for these techniques lies in what these techniques do best: finding ways to compare/score the competing predictions of theories. That's a limited role, but potentially an important one, since it is often possible to come up with contradictory interpretations of the data even over a single language when dealing with the ad hoc manual inference that linguists often use. But this limited role will be the case at least until CL/NLP returns to the old questions of symbolic representations that have largely fallen by the wayside in favor of scale. I can't say much about psycholinguistics though, but they generally already seem to use statistics.
What linguists have done is that they've found the nails first, and if you are a critic, you'd say that they've tried to bang the nails in with their fists. (If you are not a general critic of UMDian linguistics, you'd say that nails analogy is bogus, and I might actually agree with you, but that's the subject of a different post and maybe a different course.) Be that as it may, to accomplish the goal of making this class relevant to linguists, it really has to be approached from the opposite direction of an NLP applications course.
That is to say, it has to be shown that these techniques are in fact the correct set of hammers, and that means starting from the nail. The linguist's accusation against attempts at importing NLP concepts has always been that the hammer is always too big or too little. To reiterate, you can't really say, "Well, I have these techniques, they use data!!! Surely you can find something." The linguist will wonder, as linguists often do, why s/he needs more than s/he already has, and whether LSA is just too big a hammer.
Let's take syntax. A question that seems to have been a biggie for the past little while here at UMD is the "Agree" issue: do we need a special syntactic operation like "Agree"? It's still not clear that NLP techniques serve to resolve these high-level questions (eg, what operations exist and are minimally necessary). And if they don't, then one has to find lower-level questions, perhaps the mechanics of the syntax-semantics interface. But that requires NLP techniques that can map strings to at least the First Order Predicate Calculus in order to do anything at the sophistication that semanticists use.
Since data driven techniques have either sacrificed this approach out of expediency or found them impossible to support, it's hard to make the case for the involvement of NLP in at least those two domains of linguistic investigation (syntax and semantics). I would suggest that the place for these techniques lies in what these techniques do best: finding ways to compare/score the competing predictions of theories. That's a limited role, but potentially an important one, since it is often possible to come up with contradictory interpretations of the data even over a single language when dealing with the ad hoc manual inference that linguists often use. But this limited role will be the case at least until CL/NLP returns to the old questions of symbolic representations that have largely fallen by the wayside in favor of scale. I can't say much about psycholinguistics though, but they generally already seem to use statistics.
Monday, September 22, 2008
Colorless green ideas
Here is Chomsky's footnote 4 from Syntactic Structures, two pages after his introduction of the famous Colorless green/Furiously sleep pair, which, he argues, shows that "in any statistical model for grammaticalness, these sentences will be ruled out on identical grounds as equally 'remote' from English."
Even without rooting through LSLT, we see that Chomsky has immediately qualified his any statistical model, so I think any schism centring around the Colorless... argument is a result of miscommunication more than anything. This is just an argument that you can't get a blood out of a stone, and distributional information per se is to a theory of syntax a mere bloodless stone (contrary to what some people at the time apparently were proposing).
The details of his argument are not totally clear to me, but the argument about agreement seems to be an argument from nested dependencies, thus, against using a finite-state model; the argument from iterated modifiers seems to be an argument against a model in which we retrieve one of a finite set of strings of category labels, and replace each of the labels with a word. And if I've reconstructed that part of it correctly, the full argument perhaps was meant to be basically what Chris said today: if there's something fundamentally wrong with the model of language with respect to which we keep track of distributional information, then although you may get a reasonable approximation, ultimately, more data, whether you keep track of distributions or not, is not going to help. Having distributional information doesn't replace a theory of language; it presupposes one. Implicitly, one might imagine rounding the argument off by saying, Once we've come up with a proper theory of language, then maybe we can find a role for keeping track of distributions. Or maybe we won't need it.
So how does this relate to the contents of the course? Well, although it seems clear that EM is deeply important, and it's clear that finite state mappings of the kind we've been talking about are part of the language faculty, it's not clear to me that the interesting learning problem in, say, phonology, arises when we know the structure of the model and the only parameters are transition/emission probabilities; the problem is learning the structure of the machine. So these examples are admittedly a bit hard to sink my teeth into. Clustering and constructing features, however--the what's important here problems--we can easily cook up problems that look like that in any subfield of linguistics (for example, representations in both phonetics and phonology).
One might seek to develop a more elaborate relation between statistical and syntactic structure than the simple order of approximation model we have rejected. I would certainly not care to argue that any such relation is unthinkable, but I know of no suggestion to this effect that does not have obvious flaws. Notice, in particular, that for any n, we can find a string whose first n words may occur as the beginning of a grammatical sentence S1 and whose last n words may occur as the ending of some grammatical sentence S2, but where S1 must be distinct from S2. For example, consider the sequences of the form "the man who ... are here," where ... may be a verb phrase of arbitrary length. Notice also that we can have new but perfectly grammatical sequences of word classes, e.g., a sequence of adjectives longer than any ever before produced in the context "I saw a -- house." Various attempts to explain the grammatical-ungrammatical distinction, as in the case of (1), (2), on the basis of frequency of sentence type, order of approximation of word class sequences, etc., will run afoul of numerous facts like these.
Even without rooting through LSLT, we see that Chomsky has immediately qualified his any statistical model, so I think any schism centring around the Colorless... argument is a result of miscommunication more than anything. This is just an argument that you can't get a blood out of a stone, and distributional information per se is to a theory of syntax a mere bloodless stone (contrary to what some people at the time apparently were proposing).
The details of his argument are not totally clear to me, but the argument about agreement seems to be an argument from nested dependencies, thus, against using a finite-state model; the argument from iterated modifiers seems to be an argument against a model in which we retrieve one of a finite set of strings of category labels, and replace each of the labels with a word. And if I've reconstructed that part of it correctly, the full argument perhaps was meant to be basically what Chris said today: if there's something fundamentally wrong with the model of language with respect to which we keep track of distributional information, then although you may get a reasonable approximation, ultimately, more data, whether you keep track of distributions or not, is not going to help. Having distributional information doesn't replace a theory of language; it presupposes one. Implicitly, one might imagine rounding the argument off by saying, Once we've come up with a proper theory of language, then maybe we can find a role for keeping track of distributions. Or maybe we won't need it.
So how does this relate to the contents of the course? Well, although it seems clear that EM is deeply important, and it's clear that finite state mappings of the kind we've been talking about are part of the language faculty, it's not clear to me that the interesting learning problem in, say, phonology, arises when we know the structure of the model and the only parameters are transition/emission probabilities; the problem is learning the structure of the machine. So these examples are admittedly a bit hard to sink my teeth into. Clustering and constructing features, however--the what's important here problems--we can easily cook up problems that look like that in any subfield of linguistics (for example, representations in both phonetics and phonology).
Subscribe to:
Posts (Atom)