扫描手机二维码

欢迎您的访问
您是第 位访客

开通时间:..

最后更新时间:..

  • 叶昕辰 ( 副教授 )

    的个人主页 http://faculty.dlut.edu.cn/yexinchen/zh_CN/index.htm

  •   副教授   博士生导师   硕士生导师
  • 主要任职:IEEE member, ACM member
  • 其他任职:IEEE协会会员, ACM协会会员, CCF计算机协会会员
Progressive DSR (TIP'20) 当前位置: 中文主页 >> 论文及项目 >> Progressive DSR (TIP'20)




PMBANet: Progressive Multi-Branch Aggregation Network for Scene Depth Super-Resolution


Xinchen Ye1*, Baoli Sun1, Zhihui Wang*1, Jingyu Yang2, Rui Xu1, Haojie Li1, Baopu Li3

1 Dalian University of Technology, 2 Tianjin University, Baidu Research, USA

* Corresponding author

Code: https://github.com/Sunbaoli/PMBANet_DSR 

Paper: pmba-net.pdf



Abstract

Depth map super-resolution is an ill-posed inverse problem with many challenges. First, depth boundaries are generally hard to reconstruct particularly at large magnification factors. Second, depth regions on fine structures and tiny objects in the scene are destroyed seriously by downsampling degradation. To tackle these difficulties, we propose a progressive multi-branch aggregation network (PMBANet), which consists of stacked MBA blocks to fully address the above problems and progressively recover the degraded depth map. Specifically, each MBA block has multiple parallel branches: 1) The reconstruction branch is proposed based on the designed attention-based error feed-forward/-back modules, which iteratively exploits and compensates the downsampling errors to refine the depth map by imposing the attention mechanism on the module to gradually highlight the informative features at depth boundaries. 2) We formulate a separate guidance branch as prior knowledge to help to recover the depth details, in which the multi-scale branch is to learn a multi-scale representation that pays close attention at objects of different scales, while the color branch regularizes the depth map by using auxiliary color information. Then, a fusion block is introduced to adaptively fuse and select the discriminative features  from all the branches. The design methodology of our whole network is well-founded, and extensive experiments on benchmark datasets demonstrate that our method achieves superior performance in comparison with the state-of-the-art methods. 


Method

pmba-net.jpg

Figure 1.  Network architecture of the proposed PMBANet. To better present the whole framework and implementation details, different colored rectangles are

used to represent different stages and different operations in each stage.



Results


pmba-net.jpg

Figure 2. Visual comparison for recovered depth maps from 8x downsampling and noisy degradation on two examples (Books, Reindeer).


pmba-net.jpg

Figure 3.  Generalization on MPI dataset (8x cases on Alley 1-48, Ambush 4-12)


pmba-net.jpg

Figure 4.  Visualization of intermediate feature maps from all the branches in PMBANet. For clearly presenting the feature extraction process progressively,

we increase the MPA blocks up to K = 5.



Citation

Xinchen Ye, Baoli Sun, Zhihui Wang*, Jingyu Yang, Rui Xu, Haojie Li, Baopu Li, PMBANet: Progressive Multi-Branch Aggregation Network for Scene Depth Super-Resolution, IEEE Trans. Image Processing, 29:7427-7442, 2020. 

 

@article{Ye2020tip,
  author = {Xinchen Ye, Baoli Sun, Zhihui Wang, Jingyu Yang, Rui Xu, Haojie Li, Baopu Li},
  title = {PMBANet: Progressive Multi-Branch Aggregation Network for Scene Depth Super-Resolution},
  booktitle = {IEEE Trans. Image Processing (TIP)},
  year={2020}, volume={29}, pages={7427-7442},

}







辽ICP备05001357号 地址:中国·辽宁省大连市甘井子区凌工路2号 邮编:116024
版权所有:大连理工大学