Lecture 19: Structured Perceptron

Classification (eg., with decision trees or perceptron) is great when we're trying to predict a bit.  It's not so great when we're trying to predict a structured object, like a POS sequence or parse tree or Arabic sentence (eg., in translation).

The field of predicting complicated structured outputs is called one of {structured prediction, structured output prediction, predicting structured outputs, etc}.  I prefer the first term.

One of the simplest structured prediction problems is sequence labeling.  I'm given a sequence of inputs and I want to predict a corresponding sequence of outputs, where there is a one-to-one matching between inputs and outputs.  We've already seen POS tagging as an example of this task.

Another example of this task is named entity recognition.  Here, the input is a sentence, and the goal is to spot all mentions of "named entities" in the text.  Typically we also want to label them from a finite set { Person, Organization, Location, etc... }.

Since names can span more than one token (unlike POS tags), we need a way of encoding this as a sequence labeling problem.  A popular encoding (though definitely not the only one) is the BIO encoding: Begin/In/Out.  This is probably most clear by an example or two:

• Hal_B-PER is_O at_O UMD_B-ORG today_O
• Barack_B-PER Obama_I-PER is_O President_O of_O the_O United_B-GPE States_I-GPE of_I-GPE America_I-GPE
• College_B-LOC Park_I-LOC Maryland_B-LOC is_O nice_O

Here, B-X signals the start of a phrase of type X, I-X continues it, and O means not-an-entity.  Note in the last example that [College Park] and [Maryland] are different entities, so we have an I-LOC followed by a B-LOC.  Note that it's impossible to get to I-X from anything other than B-X or I-X.

Obviously we could attempt to solve this problem with an HMM.  The problem is that we might want to include a bunch of additional features about words, their contexts and their spellings, into the model.  It's difficult, if not impossible, to do this with an HMM.

So instead, we will use a structured variant of a perceptron.

The high-level idea is fairly straightforward.  Instead of extracting features over just the input, we'll extract features over the input and output together.  Call this feature vector f(x,y), where x is a complete sentence and y is a complete output.  For instance, we might have a feature: how many times is Maryland paired with B-LOC?  Or how many times is a capitalized word paired with O?  Or how many times did I see B-LOC followed by I-LOC?

The perceptron algorithm will hinge on being able to solve the "argmax" problem.  Namely, for a given x (sentence) and given weight vector w, we want to find the y that maximizes the dot product between w and f(x,y).  Note that in general this is VERY HARD.  However, if all our features obey Markov assumptions, we can use dynamic programming.

Now we can sketch the algorithm:

• Initialize w=<0 0 0 0 ... 0>
• For each intput/output example (x,y)
• Compute current prediction: y' = argmax_y w*f(x,y)
• If y == y', celebrate
• otherwise:
• Update w = w + f(x,y) - f(x,y')

That's it!

All of the nice properties of perceptron are maintained for this algorithm.  Plus, for sequence labeling, we can solve the argmax problem efficiently using a variant of the Viterbi algorithm.

If you want to know more about structured prediction, I'm giving a guest lecture in Lise Getoor's ML class on Nov 30, 3:30-4:45, in CSIC 1121.