We propose an approach for jointly filling holes and upsampling depth information for RGB-D images, where RGB color information is available at all pixel locations whereas depth information is only available at lower resolution and entirely missing in small regions referred to as “holes.” Depth information completion is formulated as a minimization of an objective function composed of two additive terms. The first data fidelity term penalizes disagreement with the observed low-resolution data. The second regularization term penalizes weighted depth deviations from a local linear model in spatial coordinates, where the weights are experimentally determined to ensure consistency between the RGB color image and the estimated depth image. We also propose a memory-efficient implementation of the proposed method based on the conjugate gradient method. Importantly, statistical analysis, which we present in this paper, also reveals that prior evaluations of depth upsampling accuracy are potentially biased because the evaluations inappropriately used preprocessed hole-filled data as “ground truth.”

• Code

We recommend using the CodeOcean version of the program, which can run using CodeOcean’s built-in interface.  You can also find the code on our GitHub page.

• Problem Formulation

The problem formulation illustrated in 1D. The magenta and cyan points show different color pixels in a color image patch, and the circles around the data points indicate the available low resolution depth values. The un-filled and filled circles indicate, respectively, input and desired depth map values, and the black line shows the weighted linear fit over the example area. The sizes of filled circles represent the weights.

• Results

Visual comparison of results obtained for 4× upsampling with different algorithms for images from the Middlebury dataset.

Visual comparison of results obtained with different algorithms for images from the ToF dataset.