Lecture 25: Mapping Text to Actions

There has been a bunch of work recently on trying to automatically find relationships between language and the "real world", where "real world" actually often means some sort of simulated environment.  Here are a few papers along these lines:

• Fleischman, M. B. and Roy, D.  Intentional Context in Situated Language Learning.  Ninth Conference on Computational Natural Language Learning , Ann Arbor, MI. June 2005.
•  Learning to Connect Language and Perception [Abstract] [PDF]
  Raymond J. Mooney
  In Proceedings of the 23rd AAAI Conference on Artificial Intelligence (AAAI), Senior Member Paper, Chicago, IL, pp. 1598-1601, July 2008.
•  S.R.K. Branavan, Harr Chen, Luke Zettlemoyer and Regina Barzilay
  "Reinforcement Learning for Mapping Instructions to Actions",
  Proceedings of ACL, 2009. Best Paper Award 
•  Learning semantic correspondences with less supervision.
  Percy Liang, Michael I. Jordan, Dan Klein.
  Association for Computational Linguistics and International Joint Conference on Natural Language Processing (ACL-IJCNLP), 2009. 
•  Adam Vogel and Dan Jurafsky. 2010. Learning to Follow Navigational Directions. In Proceedings of ACL-2010, Uppsala, Sweden. [PDF]

There are others, of course, but these five form a fairly diverse example set.  There's not much work on trying to use the real world, but robotics people like Nick Roy (at MIT) are trying to make headway on this problem.

In the first paper, which is the one we'll talk about most, the key idea is that of hierarchical plans, represented as a pcfg.  For instance we might have a rule "OfferCup -> PickUpCup MoveCup ReleaseCup", where each of the subactions might either be atomic (correspond to actual muscle movements) or might itself be broken down further.  (Qustion: how context free is this problem?)

The key ambiguity is due to the fact that actions do not select for exactly one interpretation, as in the Blicket example.

In this paper, they hand constructed a PCFG for actions and the key learning question was whether you could figure out the level of ambiguity automatically.  The basic idea is to look at relative frequencies of occurance between lexical items and nodes in the PCFG tree for the actions.