Lecture 5: N-gram models

Today we'll talk first about estimation of:

• Bernoulli random variables (coin flips)
• p(Heads) ~= Count(Heads) / Count(total flips)
• Multinomial random variables (die rolls)
• p(die comes up "4") ~= Count("4"s) / Count(total rolls)
• Now think about a giant die with |Vocabulary|-many sides!

Then about how this relates to language modeling (i.e., determining whether a sentence looks like "English" or not).

We'll then run through the chain rule to break p(s) = p(w_1 w_2 ... w_N) = prod_n p(w_n | w_{1..n-1}) and then apply a Markov assumption to the terms in the product to arrive at the n-gram language model.

This language model will be a product of Multinomial random variables, so we can apply everything we know about estimation.

Finally, we'll talk about:

• Smoothing (adding fake counts)
• Structural zeros versus statistical zeros

Where the goal of smoothing is to not think that possible things are impossible.

(Warning: the treatment we give of out of vocabulary -- "OOV" -- terms will not be very detailed!)