Lecture 18: Linear Models for Learning

Decision trees are easy to understand, but they often die because they keep splitting up the training data.

Another idea is to think of a classification as being a weighted linear combination of features.  In other words, each feature (word) gets a weight.  Highly (positive) weighted words are really indicative of class 1, and highly negative weighted words are really indicative of class -1.  We add up the weights for all the words that occur and use the sign of this as a prediction.

This is the idea behind the perceptron algorithm, which dates back to the 1960s.  It is online, which means that it processes one example at a time.  For each example, it makes a prediction.  If this prediction is right, it does nothing.  If it's wrong, it makes a small update:

• For each example (x,y):
• Compute prediction: y' = sum_{words in x} weight(word)
• If y' = y, celebrate and continue
• otherwise we've made an error:
• For each word in x:
• Set weight(word) := weight(word) + y
• (In other words, if y is negative, reduce the weight; if y is positive, increase the weight)

That's it!

It turns out that if your data is such that the there is a set of weights that would enable us to get perfect classifications on the training data, the perceptron will eventually find it!  (Of course by that time it may have overfit, so you'll probably want to stop it before it's converged.)

There is a linear algebra interpretation of the perceptron: it's optimizing a weight vector so that w*x is >0 for positive examples and w*x is <0 for negative examples.  Here, "*" means "dot product."  If you interpret dot products as projections, then it's trying to find a weight vector so that when the data points are projected onto this vector, the positive points all lie to one side, and the negative points all lie to the other side.  (Note that the usual pictures you see for this are not representative of NLP applications, where our vectors are usually corners of a very high dimensional hypercube.)

Once you think of it like this, you can think about trying to directly optimize w accoring to some loss function, which leads to a whole host of models, like linear regression, ridge regression, logistic regression, support vector machines, boosting, etc., depending on how you choose your loss function.

The basic idea is to define a cost, something like "cost = 1 if the prediction is wrong, and cost = 0 if the prediction is right."  Now, try to find a w that minimizes this cost.  Unfortunately, this cost is discrete, which makes it hard (actually NP-hard) to optimize.  So we relax it and say something like "cost gets bigger as the prediction gets more wrong" which is now continuous and easy to optimize.

The problem is that big weights are bad news: it means that our classification decision is really sensitive to just a few features.  We'd like to keep the weights small.  This thinking leads to regularization by penalizing large weights.  The standard penalization is to penalize the sum of squared weights, basically because this is continuous and differentiable.

So we're left with a cost that looks like "sum of costs on training examples + lambda * regularizer", where lambda controls the overfitting/underfitting tradeoff.