PMBANet: Progressive Multi-Branch Aggregation Network for Scene Depth Super-Resolution

Xinchen Ye^1*, Baoli Sun¹, Zhihui Wang*¹, Jingyu Yang², Rui Xu¹, Haojie Li¹, Baopu Li³

¹ Dalian University of Technology, ² Tianjin University, ³ Baidu Research, USA

* Corresponding author

Code: https://github.com/Sunbaoli/PMBANet_DSR 

Paper: pmba-net.pdf

Abstract

Depth map super-resolution is an ill-posed inverse problem with many challenges. First, depth boundaries are generally hard to reconstruct particularly at large magnification factors. Second, depth regions on fine structures and tiny objects in the scene are destroyed seriously by downsampling degradation. To tackle these difficulties, we propose a progressive multi-branch aggregation network (PMBANet), which consists of stacked MBA blocks to fully address the above problems and progressively recover the degraded depth map. Specifically, each MBA block has multiple parallel branches: 1) The reconstruction branch is proposed based on the designed attention-based error feed-forward/-back modules, which iteratively exploits and compensates the downsampling errors to refine the depth map by imposing the attention mechanism on the module to gradually highlight the informative features at depth boundaries. 2) We formulate a separate guidance branch as prior knowledge to help to recover the depth details, in which the multi-scale branch is to learn a multi-scale representation that pays close attention at objects of different scales, while the color branch regularizes the depth map by using auxiliary color information. Then, a fusion block is introduced to adaptively fuse and select the discriminative features  from all the branches. The design methodology of our whole network is well-founded, and extensive experiments on benchmark datasets demonstrate that our method achieves superior performance in comparison with the state-of-the-art methods. 

Method

Figure 1.  Network architecture of the proposed PMBANet. To better present the whole framework and implementation details, different colored rectangles are

used to represent different stages and different operations in each stage.

Results

Figure 2. Visual comparison for recovered depth maps from 8x downsampling and noisy degradation on two examples (Books, Reindeer).

Figure 3.  Generalization on MPI dataset (8x cases on Alley 1-48, Ambush 4-12)

Figure 4.  Visualization of intermediate feature maps from all the branches in PMBANet. For clearly presenting the feature extraction process progressively,

we increase the MPA blocks up to K = 5.

Citation

Xinchen Ye, Baoli Sun, Zhihui Wang*, Jingyu Yang, Rui Xu, Haojie Li, Baopu Li, PMBANet: Progressive Multi-Branch Aggregation Network for Scene Depth Super-Resolution, IEEE Trans. Image Processing, 29:7427-7442, 2020. 

 

@article{Ye2020tip,
  author = {Xinchen Ye, Baoli Sun, Zhihui Wang, Jingyu Yang, Rui Xu, Haojie Li, Baopu Li},
  title = {PMBANet: Progressive Multi-Branch Aggregation Network for Scene Depth Super-Resolution},
  booktitle = {IEEE Trans. Image Processing (TIP)},
  year={2020}, volume={29}, pages={7427-7442},

}