From virtual reality (VR) to autonomous driving, behind those advanced 5G technologies there is LiDAR sensor technology, which uses the pulse from a laser to create 3D models and maps of objects and environments. Therefore, large-scale 3D point clouds (LS3DPCs) obtained by terrestrial LiDAR scanners often include virtual points which are generated by glass reflection. Such virtual points distort the true geometry of target real-world scenes, and hence degrade the performance of the related geometry processing techniques for LS3DPCs accordingly.
A team of researchers, led by Professor Jae-Young Sim in the School of Electrical and Computer Engineering at UNIST has proposed a novel virtual point detection algorithm for LS3DPCs captured with multiple glass planes. The new technique is expected to provide an accurate way to digitize a real-world object for use in AR and VR, without any further work being required.
The image above is a LS3DPC model where the virtual points caused by glass reflection are detected in red and the associated glass plane is shown in cyan.
According to the research team, no research has been conducted to automatically remove such virtual points from LS3DPCs. Thus, the proposed method will be the first virtual point detection algorithm for LS3DPCs captured with multiple glass planes.
“LS3DPCs captured by using terrestrial LiDAR scanners also suffer from the reflection artifacts since many outdoor real-world structures include glasses, e.g., buildings,” says Jae-Seong Yun in the Combined M.S/Ph.D. of Electrical and Computer Engineering at UNIST, the first author of the study. “In contrast, using the reflective properties of glass enables obtaining accurate 3D data without removing the geometric artifacts of glass reflection in LS3DPCs.”
In the study, the team has successfully eliminated the reflection artifacts from LS3DPCs via two-step process: The first is to find multiple glass regions using the characteristics of LiDAR scanning, the second is to detect and remove the virtual points associated with multiple glass regions.
Shown above are the distribution of the number of projected points in ‘International
Hall’ model. (a) A panoramic image for target scene. (b) The distribution of the number of protected points gi. (c) The histogram of gi’s.
The glass regions can be identified by the number of laser pulses, emitted from LiDAR scanners. In general, one emitted laser is reflected only once. However, when it comes to a single glass image, it not only captures a target scene behind the glass, but also an undesired scene in front of the glass. The undesired reflected scenes usually degrade the performance of image processing and computer vision techniques when applied to glass images.
In order to remove the reflection artifacts from LS3DPC, Professor Sim and his team investigated the reflection characteristics of laser pulses emitted from a terrestrial LiDAR scanner, and estimated valid glass regions located in multiple glass planes, respectively. Then, for each point, they found all the possible trajectories of reflection, and selected the optimal trajectory by computing the validity scores combining the trajectory reliability, reflection symmetry, and geometric similarity. Thus, the researchers finally detected the virtual points associated with the optimal trajectories, which are then removed. They also captured diverse LS3DPC models with various numbers of glass planes and perform intensive experiments.
The image above is the trajectory estimation with multiple glass planes.
Experimental results demonstrated that the proposed algorithm successfully detected the virtual points associated with multiple glass planes, and removed the geometric artifacts of glass reflection in LS3DPCs. Moreover, the research team noted that the proposed algorithm yields a much better performance compared with the existing method qualitatively and quantitatively.
“From data collection to algorithm evaluation, it was challenging to proceed almost everything by ourselves. Yet, because of such experience, we were able to secure the source technology to remove the reflection artifacts in LS3DPCs,” says Professor Sim. “I hope this will help to produce high-quality 3D contents towards the realization of advanced technologies, such as autonomous driving, VR, and AR.”
The findings of this research have been published in the online version of IEEE Transactions on Pattern Analysis and Machine Intelligence on August 8, 2019. This study has been supported by the National Research Foundation of Korea (NRF) within in the Ministry of Science and ICT (MIST).
Journal Reference
Jae-Seong Yun and Jae-Young Sim, “Virtual Point Removal for Large-Scale 3D Point Clouds With Multiple Glass Planes,” IEEE TPAMI, (2019).
