Happy Holidays: a few more P4 tokens!

P4 deadline pushed back to Dec 14

The 9th is apparently the deadline for the ML project.

P4: Small error in example numbers....

There are some sentences in the training data that contain a "TAB" character.  The reasonable thing to do would be just to consider this as whitespace.  For some reason I didn't do this.  In my example of DF computation, I did this.  Which somewhat changes all the remaining numbers.

Instead of rerunning everything I'll just tell you what the updated top frequency words are if you do it "properly."  In general, for this assignment, don't worry if your numbers are slightly different than mine -- it may have to do with how you handle the non-ascii characters that appear once in a while in the data.

   2999 .
   2999 ,
   2998 of
   2997 the
   2997 and
   2994 in
   2989 to
   2988 a
   2885 as
   2875 by
   2862 for
   2860 )
   2859 (
   2836 with
   2832 that
   2801 ''
   2788 ``
   2759 on
   2717 from

P4, grading rules

So P4 has been posted for a while.  It is "optional" in the sense that your project grades will be based on your best three out of four grades.  In particular, here's what will happen.

Suppose that your grades on P1, ..., P4 are a,b,c,d.  (If you don't do P4, then d=0.)

Let x = [ (a + b + c + d) - min { a, b, c, d } ] / 3

Then x is your average grade on your best three projects.


We will use x as your overall project grade (i.e., since each project is weighed equally, it will be like you got a score of x on all FOUR of them).

P3 grading updated, deadline Wed 24th, 5p

Looks like I underestimated the difficulty of the gender classification.  I've adjusted the scoring to be easier on you.  The new scoring is:

• 35 < e < 37 : 10%
• 34 < e < 35 : 25%
• 33 < e < 34 : 32%
• 32 < e < 33 : 34%
• 31.5 < e < 32 : 36%
• 31 < e < 31.5 : 37%
• 30.5 < e < 31.0 : 38%
• 30 < e < 30.5 : 39%
• e < 30: 40%

P2 is mostly graded...

If you'd like your partial score, please send me an email with the ID of the person who handed it in.  I would love to just spam you all by sending your report to the hander-in at their id, but I don't know how to do that automatically.  I tried sending an email to myid@umd.edu but it bounced... if anyone knows how to do that instead, let me know.

Overall people seemed to do quite well.

TURKERS Final Project

Looks like there's no time tomorrow (Monday) that works for everyone.  Please vote on your time preference, trying to be as flexible as possible, and figuring that we'll spend 30 minutes.  Also, please list your email address in your signup so I can contact you!  :)

P3 is posted, due Nov 23

P2 deadline and ML midterm

I've been informed that there's a midterm in Lise's ML class on the 19th, which coincides with the P2 deadline.

To try to help you not lose too much sleep, I've pushed the deadline for P2 back to Friday night at 10p.  However, to be fair to those who don't have ML, or who have planned their schedule around the deadline on the 19th, I'll give you 15% extra credit if you hand it in by the original deadline (19th).  If you do hand it in early, please make a note in your writeup that you've done so.  Hopefully 15% is enough to balance out anything that you would lose by not being able to compete up to the last minute with your classmates on the competition parts.

Also, a note on pruning:

This is now in the .pdf writeup, but:

• Be sure that you prune all cells, including the right-most ones.  Your labeled F measure for unpruned should be about 78%.
  
• Don't forget to debinarize before you evaluate!
  
• If you prune every cell, including the root, with K=2, your labeled F measure should be about 23%.  You should have about 83 unparsable sentences.
  
• If you prune every cell except the root (which is a totally sensible thing to do -- thanks to several of you who have pointed this out -- your labeled F should be about 59%, and you should have about 38 unparsable sentences.  (As you can see, it's a really good idea not to prune the root, especially since it doesn't make things any slower -- In fact, it might actually make things faster!)
  
• You can implement it either of the above two ways -- just be sure to document which you're doing.
  
• Be sure to prune before applying unary rules.  Thus, even with K=2, you will often end up with more than 2 items in a chart cell.  However, only 2 of these will have binary derivations; all the rest should have unary derivations.

As I said in class, the danger with me trying to give you hints is that often you guys end up doing a better job implementing this stuff than I do!  But I'm pretty sure the above numbers are correct -- I've inspected my charts to make sure they look okay.

Project 2 warning on pruning results

I had a bug in my pruning, and wasn't pruning the right-most cells. (It was an off-by-one error.) Since English is mostly right-branching, the cells that touch the right edge (i.e., end of sentence) are incredibly important, so not pruning them makes life a lot better than otherwise. I'm working on fixing it right now and checking to make sure there aren't other bugs. I'll post a comment to this post when it's done and I've updated the results in the .pdf, but your pruning results should be much worse than what is in the writeup (i.e., with K=2, you should get no parse on 80-something sentences, and your f-score should be in the 10s or 20s).

Project 2 stuff...

I fixed the P2 pdf (just the one linked on the web page, not the one in the .tar.gz file) so that it includes unary productions in the examples. This is only relevant for part 2 -- nothing else changed.

Also, I'm not sure if I included them previously, but here are the links for the online test scripts:

• For playing with your hand-written grammar
• For testing your treebank parser

Projec 2 is posted, due Oct 19

It's all about parsing.  The writeup is really long, but don't be too intiminated by it.  It's long because I tried to give a lot of debugging help.  The project is all about grammars and parsing, so go off and do syntax!

Slight change to P1 "tagger" script

The old rules were that if you ran the online tagger at time X, you would have to wait until time X+24 hours to run it again.

To be nice to people who have work schedules, this has been "softened."  The new policy is that every 24 hours you earn one "token."  You spend tokens by running the tagger.  However, you can only keep a total of three tokens at any one time.

The effect of this policy is that you could continue doing one handin every 24 hours, or you could do two every 48 hours, or three every 72 hours.  But not more than that.  So on average it's the same, but you get a bit more flexibility.

The script is a bit more complicated now (the logic is harder), so please let me know if you run into any bugs.

Also, if you run the handin and just enter your ID, but not a file to upload, it will tell you how many tokens you have left and, if it's less than three, the date/times that those tokens were used so you can figure out how long until you get a new one (you get a new one when the oldest one "dies" -- i.e., 72 hours pass since the oldest one).

Hopefully this is clear :).

Project 1 is posted, due 30 Sept

Use this space to ask questions / discuss the project.

[Update 14 Sep] Several people have asked what tools/data are allowed for the rule-based tagger and for the "do as well as you can" part.  Here is an attempt to clarify.  My goal isn't to make life hard on you, but to not give unfair advantages to some subsets of the class.

• For the rule-based tagger, any rules are fair game, but it would be better if you made your tagger WITHOUT looking at the labeled training data.  That is, feel free to look at the sentences, but pretend that you don't have access to the tags.  After all, that's how it would be in the real world.  Yes, this means that you'll have to use some linguistics.  That's not a terrible thing :).
  
• For the last part of the project, where you have to build the best tagger possible, people have asked about using external tools.  For this project, I would ask that anything you use beyond normal library / unix tools stuff, you implement yourself.  Yes, you could probably do better using well-engineered tools, but that probably won't help you learn as much as you will if you do something slightly less good, but do it on your own.
  
• (Note that this final rule will not be the case for the final class project: there, anything is fair game!)