Intelligent Systems

Human-Aware Object Placement for Visual Environment Reconstruction

2022

Conference Paper

ncs

ps


Humans are in constant contact with the world as they move through it and interact with it. This contact is a vital source of information for understanding 3D humans, 3D scenes, and the interactions between them. In fact, we demonstrate that these human-scene interactions (HSIs) can be leveraged to improve the 3D reconstruction of a scene from a monocular RGB video. Our key idea is that, as a person moves through a scene and interacts with it, we accumulate HSIs across multiple input images, and optimize the 3D scene to reconstruct a consistent, physically plausible and functional 3D scene layout. Our optimization-based approach exploits three types of HSI constraints: (1) humans that move in a scene are occluded or occlude objects, thus, defining the depth ordering of the objects, (2) humans move through free space and do not interpenetrate objects, (3) when humans and objects are in contact, the contact surfaces occupy the same place in space. Using these constraints in an optimization formulation across all observations, we significantly improve the 3D scene layout reconstruction. Furthermore, we show that our scene reconstruction can be used to refine the initial 3D human pose and shape (HPS) estimation. We evaluate the 3D scene layout reconstruction and HPS estimation qualitatively and quantitatively using the PROX and PiGraphs datasets. The code and data are available for research purposes at https://mover.is.tue.mpg.de/.

Author(s): Hongwei Yi and Chun-Hao P. Huang and Dimitrios Tzionas and Muhammed Kocabas and Mohamed Hassan and Siyu Tang and Justus Thies and Michael J. Black
Book Title: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022)
Pages: 3949--3960
Year: 2022
Month: June
Publisher: IEEE

Department(s): Neural Capture and Synthesis, Perceiving Systems
Research Project(s): Capturing Contact
Bibtex Type: Conference Paper (inproceedings)
Paper Type: Conference

DOI: 10.1109/CVPR52688.2022.00393
Event Name: IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022)
Event Place: New Orleans, Louisiana

Address: Piscataway, NJ
ISBN: 978-1-6654-6947-0
State: Published

Links: project
arXiv

BibTex

@inproceedings{Yi_MOVER_2022,
  title = {Human-Aware Object Placement for Visual Environment Reconstruction},
  author = {Yi, Hongwei and Huang, Chun-Hao P. and Tzionas, Dimitrios and Kocabas, Muhammed and Hassan, Mohamed and Tang, Siyu and Thies, Justus and Black, Michael J.},
  booktitle = {2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2022)},
  pages = {3949--3960},
  publisher = {IEEE},
  address = {Piscataway, NJ},
  month = jun,
  year = {2022},
  doi = {10.1109/CVPR52688.2022.00393},
  month_numeric = {6}
}