Depth images captured by consumer depth sensors like ToF Cameras or Microsoft Kinect are often noisy and incomplete. Most existing methods recover missing depth values from low quality measurements using information in the corresponding color images. However, the performance of such methods is susceptible when color image is noisy or correlation between RGB-D is weak. This paper presents a depth map enhancement algorithm based on Riemannian Geometry that performs depth map de-noising and completion simultaneously. The algorithm is based on the observation that similar RGB-D patches lie in a very low-dimensional subspace over the Riemannian quotient manifold of varying-rank matrices. The similar RGB-D patches are assembled into a matrix and optimization is performed on the search space of this quotient manifold with Kronecker product trace norm penalty. The proposed convex optimization problem on a special quotient manifold essentially captures the underlying structure in the color and depth patches. This enables robust depth refinement against noise or weak correlation between RGB-D data. This non-Euclidean approach with Kronecker product trace-norm constraints and cones in the non-linear matrix spaces provide a proper geometric framework to perform optimization. This formulates depth map enhancement as a matrix completion problem in the product space of Riemannian manifolds. This Riemannian submersion automatically handles ranks that change over matrices, and ensures guaranteed convergence over constructed manifold. The experiments on public benchmarks RGB-D images show that proposed method can effectively enhance depth maps. © 2018 ACM.