PLA: Language-Driven Open-Vocabulary 3D Scene Understanding

Runyu Ding1*, Jihan Yang1*, Chuhui Xue2, Wenqing Zhang2, Song Bai2†, Xiaojuan Qi1†,
1The University of Hong Kong, 2ByteDance Inc.
CVPR 2023
*Indicates Equal Contribution, † indicates Equal Corresponding
Paper arXiv Code
Synonymical novel concepts: "couch", "freezer" Hierarchical novel concepts: "bathroom", "kitchen" Fine-grained novel concepts: "monitor", "blackboard"

Abstract

Open-vocabulary scene understanding aims to localize and recognize unseen categories beyond the annotated label space. The recent breakthrough of 2D open-vocabulary perception is largely driven by Internet-scale paired image-text data with rich vocabulary concepts. However, this success cannot be directly transferred to 3D scenarios due to the inaccessibility of large-scale 3D-text pairs. To this end, we propose to distill knowledge encoded in pre-trained vision-language (VL) foundation models through captioning multi-view images from 3D, which allows explicitly associating 3D and semantic-rich captions. Further, to facilitate coarse-to-fine visual-semantic representation learning from captions, we design hierarchical 3D-caption pairs, leveraging geometric constraints between 3D scenes and multi-view images. Finally, by employing contrastive learning, the model learns language-aware embeddings that connect 3D and text for open-vocabulary tasks. Our method not only remarkably outperforms baseline methods by 25.8% ~ 44.7% hIoU and 14.5% ~ 50.4% hAP50 on open-vocabulary semantic and instance segmentation, but also shows robust transferability on challenging zero-shot domain transfer tasks.

Approach

Image-bridged Point-language Association

We utilize multi-view images of a 3D scene as a bridge to access knowledge encoded in vision-language models. Text discritions are first generated by a powerful image-captioning model, and is then associated with a set of points in the 3D scene with geometry constraints between images and 3D point clouds. We present hierarchical scene-level, view-level and entity-level point-language association manners to elicit coarse-to-fine semantic-rich language supervision.

Langauge-driven 3D Scene Understanding Framework

Different from the close-set network, the learnable semantic head is replaced by category embeddings encoded by a text encoder from category names. Binary head is to rectify semantic scores with base and novel probability as condition. Instance head is tailored to instance segmentation. Most importantly, to endow the model with rich semantic space to improve open-vocabulary capability, we supervise point embeddings with caption embeddings based on point-language association.

Visualizations

Hierarchical Point-language Association

BibTeX

@inproceedings{ding2022language,
    title={PLA: Language-Driven Open-Vocabulary 3D Scene Understanding},
    author={Ding, Runyu and Yang, Jihan and Xue, Chuhui and Zhang, Wenqing and Bai, Song and Qi, Xiaojuan},
    booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
    year={2023}
}