摘要
This paper addresses the problem of the semantic segmentation of large-scale 3D road scenes by incorporating the complementary advantages of point clouds and images.To make full use of geometrical and visual information,this paper extracts 3D geometric features from a point cloud using a deep neural network for 3D semantic segmentation and extracts 2D visual features from images using a Convolutional Neural Network(CNN)for 2D semantic segmentation.In order to bridge the features of the two modalities,this paper uses superpoints as an intermediate representation to connect the 2D features with the 3D features.A superpoint-based pooling method is proposed to fuse the features from the two different modalities for joint learning.To evaluate the approach,the paper generates 3D scenes from the Virtual KITTI dataset.The results of the experiments demonstrate that the proposed approach is capable of segmenting large-scale 3D road scenes based on the compact and semantically homogeneous superpoints,and that it achieves considerable improvements over the 2D image and 3D point cloud semantic segmentation methods.
This paper addresses the problem of the semantic segmentation of large-scale 3D road scenes by incorporating the complementary advantages of point clouds and images.To make full use of geometrical and visual information, this paper extracts 3D geometric features from a point cloud using a deep neural network for 3D semantic segmentation and extracts 2D visual features from images using a Convolutional Neural Network(CNN)for 2D semantic segmentation.In order to bridge the features of the two modalities, this paper uses superpoints as an intermediate representation to connect the 2D features with the 3D features.A superpoint-based pooling method is proposed to fuse the features from the two different modalities for joint learning.To evaluate the approach, the paper generates 3D scenes from the Virtual KITTI dataset.The results of the experiments demonstrate that the proposed approach is capable of segmenting large-scale 3D road scenes based on the compact and semantically homogeneous superpoints, and that it achieves considerable improvements over the 2D image and 3D point cloud semantic segmentation methods.
基金
supported by the National Natural Science Foundation of China(No.U1764264/61873165)
Shanghai Automotive Industry Science and Technology Development Foundation(No.1807)
the International Chair on Automated Driving of Ground Vehicle.
作者简介
Liuyuan Deng,received the BS degree from the University of Electronic Science and Technology of China,Chengdu,China,in 2013.He is currently a PhD candidate at the Department of Automation,Shanghai Jiao Tong University,Shanghai,China.His research interests include computer vision,deep learning,semantic scene understanding,and visual localization for intelligent driving.E-mail:lydeng@sjtu.edu.cn;Corresponding author:Ming Yang,received the MS and PhD degrees from Tsinghua University,Beijing,China,in 1999 and 2003,respectively.He is currently a full tenure professor at Shanghai Jiao Tong University,director of the Department of Automation,and the deputy director of the Innovation Center of Intelligent Connected Vehicles.He has been working in the field of intelligent vehicles for more than 20 years.E-mail:MingYang@sjtu.edu.cn;Zhidong Liang,is currently a master student in Shanghai Jiao Tong University.He received the BS degree from the same institution in 2017.His research interests include deep learning,3D semantic instance segmentation,and autonomous driving.Email:709800954@qq.com;Yuesheng He,received the PhD degree from the Department of Computer Science,Hong Kong Baptist University,Hong Kong,in 2012.He held a post-doctoral position at the Department of Computer and Information Science,Faculty of Science and Technology,University of Macao.He is currently a research fellow with the Department of Automation,Shanghai Jiao Tong University.His research areas are machine learning,computer graphics,virtual reality,computer 3D animation,computer image,and signal processing.E-mail:heyuesh@sjtu.edu.cn;Chunxiang Wang,received the PhD degree from Harbin Institute of Technology,China,in 1999.She is currently an associate professor with the Department of Automation,Shanghai Jiao Tong University,Shanghai,China.Her research interests include autonomous driving,assistant driving,and mobile robots,etc.She has been working in the field of intelligent vehicles for more than 10 years and participated in several related research projects,such as European CyberC3 project,ITER transfer cask project,etc.E-mail:wangcx@sjtu.edu.cn
