摘要
Significant progress has been made in image inpainting methods in recent years.However,they are incapable of producing inpainting results with reasonable structures,rich detail,and sharpness at the same time.In this paper,we propose the Pyramid-VAE-GAN network for image inpainting to address this limitation.Our network is built on a variational autoencoder(VAE)backbone that encodes high-level latent variables to represent complicated high-dimensional prior distributions of images.The prior assists in reconstructing reasonable structures when inpainting.We also adopt a pyramid structure in our model to maintain rich detail in low-level latent variables.To avoid the usual incompatibility of requiring both reasonable structures and rich detail,we propose a novel cross-layer latent variable transfer module.This transfers information about long-range structures contained in high-level latent variables to low-level latent variables representing more detailed information.We further use adversarial training to select the most reasonable results and to improve the sharpness of the images.Extensive experimental results on multiple datasets demonstrate the superiority of our method.Our code is available at https://github.com/thy960112/Pyramid-VAE-GAN.
基金
The authors gratefully acknowledge the financial support of the National Natural Science Foundation of China(Grant No.61925603).
作者简介
Huiyuan Tian received her bachelor degree from Northwestern Polytechnic University in 2016.She is currently pursuing a Ph.D.degree in the College of Computer Science and Technology,Zhejiang University,Hangzhou,China.Her current research interests include computer vision,machine learning,and probabilistic graphical models.E-mail:,tianhuiyuan@zju.edu.cn;Li Zhang received his B.Eng.and Ph.D.degrees from Zhejiang University,in 2007 and 2013,respectively.He is currently an assistant researcher in the Department of Computer Science,Zhejiang University.In 2009,he was a visiting scholar at the University of Hong Kong.From 2013 to 2017,he was a researcher in Works Applications Co.,Ltd.His current interests include deep learning,game theory,human-machine hybrid computing,and pervasive computing.E-mail:zhangli85@zju.edu.cn;Shijian Li received his Ph.D.degree from Zhejiang University in 2006.In 2010,he was a visiting scholar with the Institute Telecom SudParis,France.He currently works in the College of Computer Science and Technology,Zhejiang University.His research interests include sensor networks,ubiquitous computing,and social computing.He serves as an Editor of the International Journal of Distributed Sensor Networks.E-mail:shijianli@zju.edu.cn;Min Yao received his Ph.D.degree in biomedical engineering and instruments from Zhejiang University in 1995.He is currently a professor in the College of Computer Science and Technology,Zhejiang University.His research interests include computational intelligence,pattern recognition,knowledge discovery,and knowledge services.E-mail:myao@zju.edu.cn;Gang Pan received his B.Eng.and Ph.D.degrees from Zhejiang University,in 1998 and 2004,respectively.He is currently a professor in the Department of Computer Science,and deputy director of the State Key Lab of CAD&CG,Zhejiang University,China.His current interests include artificial intelligence,pervasive computing,brain-inspired computing,and brain-machine interfaces.He serves as an Associate Editor of IEEE Trans.Neural Networks and Learning Systems,IEEE Systems Journal,and Pervasive and Mobile Computing.E-mail:gpan@zju.edu.cn。
