SceneCtrl: Mixed Reality Enhancement via Efficient Scene Editing

Author List

Ya-Ting Yue

The University of Hong Kong

Yong-Liang Yang

University of Bath

Gang Ren

Xiamen University of Technology

Wenping Wang

The University of Hong Kong

Project Introduction


Due to the development of 3D sensing and modeling techniques, the state-of-the-art mixed reality devices such as Microsoft Hololens have the ability of digitalizing the physical world. This unique feature bridges the gap between virtuality and reality and largely elevates the user experience. Unfortunately, the current solution only performs well if the virtual contents complement the real scene. It can easily cause visual artifacts when the reality needs to be modified due to the virtuality (e.g., remove real objects to offer more space for virtual models), a common requirement in mixed reality applications such as room redecoration and environment design. We present a novel system, called SceneCtrl, that allows the user to interactively edit the real scene sensed by Hololens, such that the reality can be adapted to suit virtuality. Our proof-of-concept prototype employs scene reconstruction and understanding to enable efficient editing such as deleting, moving, and copying real objects in the scene. We also demonstrate SceneCtrl on a number of example scenarios in mixed reality, verifying that enhanced experience resolves conflicts between virtuality and reality.

3D Beach Art | Meghna


We thank all the volunteers for participation, reviewers for helpful comments. This work was supported in part by EPSRC Grant (EP/M023281/1), Fujian Provincial Social Science Project (FJ2016C095), Xiamen Overseas Scholar Project (XRS2016 314-10), and Fujian Provinci al Natural Science Project (2017J01784).


author = {Ya-Ting Yue, Yong-Liang Yang, Gang Ren, Wenping Wang},
title = {SceneCtrl: Mixed Reality Enhancement via Efficient Scene Editing},
booktitle = {Proceedings of the 30th Annual Symposium on User Interface Software and Technology},
series = {UIST '17},
year = {2017},
pages = {},