Fuxin Li, Oregon State University, 'Some New Designs of Convolutional and Recurrent Networks'

Friday, April 23, 2021 - 2:40pm to 3:40pm
Zoom - contact events@cs.stonybrook.edu for Zoom info.
Event Description: 

Abstract: Convolutional and recurrent networks have driven the success of deep learning for the past decade. However, their limitations are starting to surface, such as convolutional networks (CNNs) relying on a fixed grid and recurrent networks having trouble handling long term memory. In this talk, we talk about some of our recent explorations into the fundamental designs of these architectures. We'll start with PointConv, which efficiently implements CNN on irregularly spaced point cloud data. This allows us to perform convolution in any arbitrary set of data, as long as they are located in a low-dimensional space. This includes traditional images, 3D point clouds, cost volumes for non-rigid point matching, as well as other sensors irregularly placed in the 3D space. Moreover, it allows the incorporation of scale and rotation invariance into the CNN with minimal effort. We show the utility of PointConv in both 2D images such as CIFAR-10 and ImageNet, to state-of-the-art performances on 3D semantic segmentation problems such as ScanNet and SemanticKITTI.

On the temporal domain, I will talk about our experience building a novel framework with a bilinear, multi-slot memory system that are more suitable to memorize multiple appearance templates for each object. Compared with the popular transformers, our memory system is constant-size, which make it more suitable to handle long video sequences. Results on the MOT 2016 and MOT 2017 challenges show that it significantly outperform traditional LSTMs in terms of identity switches, helps us to achieve real-time, online tracking with state-of-the-art performance.

Our work places the cornerstones of an object-oriented system of understanding the spatio-temporal world. An important aspect of the understanding is to correctly predict the uncertainty of the future states. If time allows, we will talk briefly about our recent work on Bayesian deep learning, where we train a generative network that generates a posterior distribution of neural networks for particle-based variational inference. This has utilities in many problems, including uncertainty estimation, outlier detection and exploration in reinforcement learning.

Bio: Fuxin Li is currently an assistant professor in the School of Electrical Engineering and Computer Science at Oregon State University. Before that, he has held research positions in University of Bonn and Georgia Institute of Technology. He had obtained a Ph.D. degree in the Institute of Automation, Chinese Academy of Sciences in 2009. He has won an NSF CAREER award, an Amazon Research Award, (co-)won the PASCAL VOC semantic segmentation challenges from 2009-2012, and led a team to the 4th place finish in the DAVIS Video Segmentation challenge 2017. He has published more than 60 papers in computer vision, machine learning and natural language processing. His main research interests are deep learning, video object segmentation, multi-target tracking, point cloud deep networks, uncertainty estimation in deep learning and human understanding of deep learning.

Computed Event Type: 
Event Title: 
Some New Designs of Convolutional and Recurrent Networks