Lecture 20: World Knowledge from Text

Today we'll talk about the DIRT algorithm and related ideas.  There are actually tons of names for problems very similar to what we're looking at: discovering inference rules, (automatic) paraphrasing, semantic parsing, etc.  (The last one is a bit of a stretch, IMO.)

But the whole idea is to identify cases of substitutability.  For example, knowing that "X wrote Y" means roughly the same thing as "X is the author of Y", we can potentially do a better job answering questions like "Who wrote the Star Spangled Banner?"

The main "magic" we'll employ to get this to work is the Distributional Hypothesis, which is one of those things that no one cites anymore because everyone knows it.  (The citation is: Harris, Z. 1985. Distributional Structure. In: Katz, J. J. (ed.) The Philosophy of Linguistics. New York: Oxford University Press. pp. 26-47.... Zellig Harris is a famous linguist!)  DH states that words in similar contexts tend to have similar meanings, and is often paraphrased as "you shall know a word by its friends."

Before talking about what DIRT does, we can talk about previous approaches that take a very "neighborly" approach to similarity.

• For each word w in your vocabulary (eg., words that appear >= 100 times)
• Find all contexts in which w appears, where a context is just two words to the left and two words to the right.
• For each context, a b w c d, increment counters as follows:
• c(w,a,left2) ++
• c(w,b,left1) ++
• c(w,c,right1) ++
• c(w,d,right2) ++
• Do this for all words and all counters
• Now, we have a representation of each word in terms of its contexts
• Define the similarity between two words w and w' as, for example, the cosine similarity between their context representations:
• sim(w,w') = <c(w), c(w')> / ( ||c(w)|| ||c(w')|| )
• where <x,y> means dot product, ||x|| means norm (= sqrt(sum_i x_i^2))
• and c(w) is the context vector obtained by looking at all c(w,?,?)
• Say that two words are similar if their similarity crosses some threshold

In DIRT, they use a syntactic notion of similarity, rather than a proximity notion of similarity.  In order to do so, they leverage dependency trees as a representation of syntax.  A dependency tree is another representation of syntax (that we didn't talk about) where you have words as nodes in a graph, and edges point from a word to its closest dependents.  You can get from a constituency tree to a dependency tree by having each head (i.e., a V is the head of a VP) point to all of its arguments and adjuncts.  But it's easier to get dependency trees directly (Google for MSTParser or MaltParser).

Once we have a dependency tree, the idea is very similar to what we talked about in semantic role labeling: pick two words in a sentence and look at the path that connects them.  There's a good example in the paper on p2.

The DIRT algorithm now works by a process similar to the above.

• For each dependency path p from w to w' in a sentence
• Increment the count of (p, X, w) and (p, Y, w'), indicating that w is playing the role of X in "X wrote Y" and w' is playing the role of Y
• Compute the similarity between any two paths by Eqs (1), (2) and (3) in the paper.  Basically first compute similarities between slots and fillers, then between slots and slots, and then between paths and paths.
• Do some heuristicy things to make it run efficiently.