Depth Super-Resolution via Deep Controllable Slicing Network 

Xinchen Ye^1*, Baoli Sun¹, Zhihui Wang*¹, Jingyu Yang², Rui Xu¹, Haojie Li¹, Baopu Li³

¹ Dalian University of Technology, ² Tianjin University, ³ Baidu Research, USA

* Corresponding author

Paper: DepthSlice.pdf

Abstract

Due to the imaging limitation of depth sensors, high-resolution (HR) depth maps are often difficult to be acquired directly, thus effective depth super-resolution (DSR) algorithms are needed to generate HR output from its low-resolution (LR) counterpart. Previous methods treat all depth regions equally without considering different extents of degradation at region-level, and regard DSR under different scales as independent tasks without considering the modeling of different scales, which impede further performance improvement and practical use of DSR. To alleviate these problems, we propose a deep controllable slicing network from a novel perspective. Specifically, our model is to learn a set of slicing branches in a divide-and-conquer manner, parameterized by a distance-aware weighting scheme to adaptively aggregate different depths in an ensemble. Each branch that specifies a depth slice (e.g., the region in some depth range) tends to yield accurate depth recovery. Meanwhile, a scale-controllable module that extracts depth features under different scales is proposed and inserted into the front of slicing network, and enables finely-grained control of the depth restoration results of slicing network with a scale hyper-parameter. Extensive experiments on synthetic and real-world benchmark datasets demonstrate that our method achieves superior performance.

Method

Figure 1.  Overview of the network architecture. SCM is to realize DSR with different downscaling factors in an unified model, which allows to finely-grained control the depth restoration results by using a scale parameter, while DSM aims to learn a set of slicing branches in a divide-and-conquer manner, parameterized by a distance-aware weighting scheme to adaptively aggregate all the branches in the ensemble.

Figure 2.  Network architecture of our SCM. GB aims to extract the common features from the input, while SB takes the given scale parameter and its corresponding LR depth map as input, then generates specialized features to adaptively tune the features of GB through FB in a linear fusion fashion.

Results

Figure 3. Visual comparison for recovered depth maps from ×8 downsampling on NYU v2 dataset.

Figure 4. Visual comparison of ×8 (𝛾𝑖𝑛 = 3) upsampling results at different 𝛾𝑖𝑛 vaules.

Figure 5. Visualization of the weighting masks.

Citation

Xinchen Ye*, Baoli Sun, Zhihui Wang, Jingyu Yang, Rui Xu, Haojie Li, Baopu Li, Depth Super-Resolution via Deep Controllable Slicing Network, ACM International Conference on Multimedia (ACMMM), 2020, Seattle, USA. 

 

@article{Ye2020acmmm,
  author = {Xinchen Ye, Baoli Sun, Zhihui Wang, Jingyu Yang, Rui Xu, Haojie Li, Baopu Li},
  title = {Depth Super-Resolution via Deep Controllable Slicing Network},
  booktitle = {ACM International Conference on Multimedia (ACMMM)},
  year={2020},

}