We propose an approach for jointly filling holes and upsampling depth information for RGB-D images, where RGB color information is available at all pixel locations whereas depth information is only available at lower resolution and entirely missing in small regions referred to as “holes.” Depth information completion is formulated as a minimization of an objective function composed of two additive terms. The first data fidelity term penalizes disagreement with the observed low-resolution data. The second regularization term penalizes weighted depth deviations from a local linear model in spatial coordinates, where the weights are experimentally determined to ensure consistency between the RGB color image and the estimated depth image. We also propose a memory-efficient implementation of the proposed method based on the conjugate gradient method. Importantly, statistical analysis, which we present in this paper, also reveals that prior evaluations of depth upsampling accuracy are potentially biased because the evaluations inappropriately used preprocessed hole-filled data as “ground truth.”
We recommend using the CodeOcean version of the program, which can run using CodeOcean’s built-in interface. You can also find the code on our GitHub page.
Local-linear-fitting-based matting for joint hole filling and depth upsampling of RGB-D images Journal Article
In: J. Electronic Imaging, vol. 28, no. 3, pp. 033019-1 – 13, 2019, (code available on Code Ocean (https://doi.org/10.24433/CO.5593522.v1)).
A local-linear-fitting-based matting approach for accurate depth upsampling Inproceedings
In: Proc. IEEE Western NY Image and Signal Proc. Wksp. (WNYISPW), pp. 1-5, Rochester, NY, 2016, (Best Paper Award).