Lecture 10: Statistical Parsing

Context free grammars are great, but as with any X, if X is great, then probabilistic X is greater still!

A probabilistic context free grammar (PCFG) is a CFG with "weights" attached to each of the rules.  These weights must be non-negative.  For any left-hand-side, the sum of weights must be one.  The following is a small PCFG:

S   0.5 -> NP VP
S   0.5 -> VP
NP  0.7 -> Det Noun
NP  0.2 -> Proper
NP  0.1 -> NP PP
VP  0.6 -> Verb NP
VP  0.3 -> Verb
VP  0.1 -> VP PP

Note that for any given LHS (eg., "NP") the sum of weights (probabilities) is one.

The probability of a tree is just the product of probabilities of rules used in that tree.

We can generate random sentences with a PCFG by starting at "S" and then picking a rule at random (weighted by the probabilities) and then recursively generating the corresponding children.

The probability of a sentence under this grammar is obtained by summing (marginalizing) out all the trees that can generate that sentence.

To parse with a PCFG, we just parse with the corresponding CFG, but then weigh items in the chart by their probabilities.  If we find a higher-probability derivation of the same item in the chart, we throw out the old one and replace it with the new one.  In CKY this is easy.

A treebank is a collection of human-labeled trees.  Given a treebank, we can extract the corresponding PCFG by counting productions and normalizing.  Most trees are not binary so you need to binarize your rules.

Note on smoothing: you have to smooth if you want your grammar to be able to parse out-of-sample trees.  Smoothing can be done just like in HMMs, but you want to be careful not to compute stuff that is irrelevant.

Note on efficiency: The grammars we produce are huge (tens or hundreds of thousands of rules).  As a result, paying O(N^3|G|) is often prohibitive, even for N around 30 or 40.  A simple solution is pruning: after you process each cell in the chart, throw out it's low probability items.  You can either do count-based pruning (only keep the top 5 in each cell), ratio-based pruning (only keep the ones within a factor of 10 probability of the best one), or probability-based pruning (only keep the ones with probability at least 10e-6).  Usually the last sucks, but the first two work reasonably well.