Lecture 17: Classification with Decision Trees

Machine learning is all about predicting the future given the past.

A particular type of machine learning problem that crops up in all sorts of NLP applications is classification.  We get an object and try to put it into one of K buckets.  For example, spam filtering (K=2) or sentiment detection (K=2) or news versus sports versus entertainment (K=3).  When K=2 it's called binary classification.  Otherwise, multiclass.

The general setup is:

• Someone gives us a bunch of examples (x) together with their correct labels (y) -- this is our training data.
• We learn some function h (for "hypothesis", maybe I'll call it f) that maps x to y
• Then at "test time" someone gives us a bunch more (x) and we have to predict the corresponding (y)s
• We are evaluated based on how accurate we were at this prediction

Key question #1: Why don't I just store the training data in a database and perform query lookups?

Key question #2: Why don't I just figure out what the most popular class (y) is and always report that?

Bad news: In general, in theory, learning is impossible.
Good news: In general, in practice, learning is EASY and works even when you might not expect it to.

Key concept is a feature vector.  We take our object and extract features of that object, which is the representation that the learning will use.  Good "feature engineering" is often worth more than good machine learning.  (I say that even as a machine learning person!)

A very simple learning algorithm is a decision tree. It's like playing "20 questions." The idea is to pretend that you have data, and get to pick only one feature to look at.  You pick that feature (let's pretend it's binary) and then split your data based on that feature.  We now have two data sets.  For each of these, we recurse and pick a new feature (it need not be the same on the two branches).  At the end, we'll be down to a "leaf" (only one data point left) in which case we don't need to ask any more questions.  (Or we'll have run out of features.)

Running the decision tree all the way to the leaves is like storing the examples in a database (why?).  Running a zero-level decision tree is like remembering the most popular class (why?).

In general, we want something in between, and to choose the depth heuristically.

To do that, we'll set aside a bit of our training data as "development data" (or "validation data") in order to pick the "best" depth.  (Or other stopping criteria.)  Important point: why can't I just use the training data?