Feature representation of RGB-D images using joint spatial-depth feature pooling

TY - JOUR

T1 - Feature representation of RGB-D images using joint spatial-depth feature pooling

AU - Pan, Hong

AU - Olsen, Søren Ingvor

AU - Zhu, Yaping

PY - 2016/9/1

Y1 - 2016/9/1

N2 - Recent development in depth imaging technology makes acquisition of depth information easier. With the additional depth cue, RGB-D cameras can provide effective support for many RGB-D perception tasks beyond traditional RGB information. However, current feature representation based on RGB-D images utilizes depth information only to extract local features, without considering it to improve robustness and discriminability of the feature representation by merging depth cues into feature pooling. Spatial pyramid model (SPM) has become the standard protocol to split a 2D image plane into sub-regions for feature pooling of RGB-D images. We argue that SPM may not be the optimal pooling scheme for RGB-D images, as it only pools features spatially and completely discards their depth topological structures. Instead, we propose a novel joint spatial-depth pooling (JSDP) scheme which further partitions SPM using the depth cue and pools features simultaneously in 2D image plane and along the depth direction. By combining the JSDP with standard feature extraction and feature encoding modules, we outperform state-of-the-art methods on benchmarks for RGB-D object classification, detection and scene recognition.

AB - Recent development in depth imaging technology makes acquisition of depth information easier. With the additional depth cue, RGB-D cameras can provide effective support for many RGB-D perception tasks beyond traditional RGB information. However, current feature representation based on RGB-D images utilizes depth information only to extract local features, without considering it to improve robustness and discriminability of the feature representation by merging depth cues into feature pooling. Spatial pyramid model (SPM) has become the standard protocol to split a 2D image plane into sub-regions for feature pooling of RGB-D images. We argue that SPM may not be the optimal pooling scheme for RGB-D images, as it only pools features spatially and completely discards their depth topological structures. Instead, we propose a novel joint spatial-depth pooling (JSDP) scheme which further partitions SPM using the depth cue and pools features simultaneously in 2D image plane and along the depth direction. By combining the JSDP with standard feature extraction and feature encoding modules, we outperform state-of-the-art methods on benchmarks for RGB-D object classification, detection and scene recognition.

KW - Faculty of Science

KW - Computer Science

KW - Image analysis

KW - Computer Vision

KW - RGB-D feature representation

U2 - 10.1016/j.patrec.2016.04.001

DO - 10.1016/j.patrec.2016.04.001

M3 - Journal article

SN - 0167-8655

VL - 80

SP - 239

EP - 248

JO - Pattern Recognition Letters

JF - Pattern Recognition Letters

IS - 1

ER -