TDAI's Foundations of Data Science & AI community of practice is pleased to present Dr. Xin (Eric) Wang (Computer Science & Engineering, University of California, Santa Cruz) on Monday, Feb. 5, at 10:00 a.m. in 301 Pomerene Hall, who will give a talk entitled "Multimodal Embodied Agents: Situated Vision, Language, and Action Learning."
Register for in-person Register for Zoom
Abstract:
A key ambition in AI research is to develop multimodal embodied agents that can interact with humans, understand their environment, and perform tasks in the real world. These agents have the potential to transform a range of applications, from household chores to critical missions.
In this talk, I will present our latest breakthroughs in this domain, addressing key challenges of situated decision-making. We begin by exploring the extension of large language models (LLMs) for commonsense reasoning in the fundamental object navigation task, by calibrating the probabilistic outputs of LLMs in a “soft” manner.
Then I will discuss the significance of vision and language grounding in embodied agents, and introduce our pioneering method that employs a causal lens to improve this grounding capability, which has shown remarkable impacts in various downstream tasks. In addition, I will briefly overview our comprehensive efforts in advancing and evaluating these agents across different scenarios, and conclude with a discussion of future research plans.
Bio:
Xin (Eric) Wang is an Assistant Professor of Computer Science and Engineering at UC Santa Cruz. His research interests include Natural Language Processing, Computer Vision, and Machine Learning, with an emphasis on Multimodal and Embodied AI. He worked at Google AI, Facebook AI Research, Microsoft Research, and Adobe Research.
Xin has served as Area Chair for conferences such as ACL, NAACL, EMNLP, ICLR, and NeurIPS, as well as a Senior Program Committee for AAAI and IJCAI. He organized workshops and tutorials at conferences such as ACL, NAACL, CVPR, and ICCV. He has received several awards and recognitions for his work, including CVPR Best Student Paper Award, Google Research Faculty Award, Amazon Alexa Prize Awards, and various gift awards from Adobe, Snap, eBay, etc.
