The first talk of our NLP Speaker Series is taking place April 28 with Yoav Artzi, assistant professor at Cornell University. Artzi works in the intersection of NLP, machine learning, vision, and robotics. His recent work won the best paper award at EMNLP 2015, a premier NLP conference. His talk titled “Mapping instructions to actions” will describe grounding language using techniques from NLP, vision, and reinforcement learning.

Everyone is welcome to attend. CSE 600 students: please remember to sign in. 

When: April 28, 2017 at 2 PM

Where: NCS 120

Title: Mapping instructions to actions

Abstract: 

An agent following instructions requires a robust understanding of language and its environment. In this talk I will describe two approaches to address the problem of mapping instructions to actions. First, I will describe a semantic parsing approach, where language is mapped to an intermediate formal representation. This method allows to explicitly model context-dependent phenomena, observe what representations the system recovers, and leverage expert knowledge. In the second part, I will propose a neural network method that jointly reasons about instructions and raw visual input obtained from a camera sensor. This approach does not require intermediate representations, planning procedures, or training different models for visual and language reasoning. Training uses reinforcement learning in a few-samples regime with reward shaping to exploit training data. While the two approaches address a similar problem, they pose different challenges and provide different advantages.

Bio: Yoav Artzi is an Assistant Professor in the Department of Computer Science and Cornell Tech at Cornell University. His research focuses on learning expressive models for natural language understanding, most recently in situated interactive scenarios. He received the best paper award in EMNLP 2015 and a Google faculty award. Yoav holds a B.Sc. summa cum laude from Tel Aviv University and a Ph.D. from the University of Washington.