Classification Algorithms

This work has been supported by the following NSF grants: IIS-0915971, CCF-0916452 and MRI CNS-0821384. This work has been also be supported by the Google Research Award "Classification of vehicles in points clouds of urban scenes, as well as from an NVIDIA equipment grant. Finally, support has been provided by CUNY PSC-CUNY and Bridge funds.

CNN-based algorithms



We examine the task of point-level object segmentation in outdoor urban LIDAR scans. A key challenge in this area is the problem of missing points in the scans due to technical limitations of the LIDAR sensors. Our core contributions are demonstrating the benefit of reframing the segmentation task over the scan acquisition grid as opposed to considering only the acquired 3D point cloud and developing a pipeline for training and applying a convolutional neural network to accomplish this segmentation on large scale LIDAR scenes. By labeling missing points in the scanning grid we show that we can train our classifier to achieve a more accurate and complete segmentation mask for the vehicle object category which is particularly prone to missing points. Additionally we show that the choice of input features maps to the CNN significantly effect the accuracy of the segmentation and these features should be chosen to fully encapsulate the 3D scene structure. We evaluate our model on a LIDAR dataset collected by Google Street View cars over a large area of New York City.

Algorithms based on parts



Unprecedented amounts of 3D data can be acquired in urban environments, but their use for scene understanding is challenging due to varying data resolution and variability of objects in the same class. An additional challenge is due to the nature of the point clouds themselves, since they lack detailed geometric or semantic information that would aid scene understanding. In this paper we present a general algorithm for segmenting and jointly classifying object parts and the object itself. Our pipeline consists of local feature extraction, robust RANSAC part segmentation, partlevel feature extraction, a structured model for parts in objects, and classification using state-of-the-art classifiers. We have tested this pipeline in a very challenging dataset that consists of real world scans of vehicles. Our contributions include the development of a segmentation and classification pipeline for objects and their parts; and a method for segmentation that is robust to the complexity of unstructured 3D points clouds, as well as a part ordering strategy for the sequential structured model and a joint feature representation between object parts.


Online Algorithms


Sequential Classification in Point Clouds of Urban Scenes


Laser range scanners have now the ability to acquire millions of 3D points of highly detailed and geometrically complex urban sites, opening new avenues of exploration in modeling urban environments. In the traditional modeling pipeline, range scans are processed off-line after acquisition. The slow sequential acquisition though is a bottleneck. The goal of our work is to alleviate this bottleneck, by exploiting the sequential nature of the data acquisition process. We have developed novel online algorithms, never before used in laser range scanning, that perform data classification on-the-fly as data is being acquired. These algorithms are extremely efficient, and can be potentially integrated with the scanner’s hardware, rendering a sensor that not only acquires but also intelligently processes and classifies the scene points. This sensor, armed with the proposed algorithms, can classify 3D points in real-time as being in vegetation vs. non-vegetation regions, or in horizontal vs. vertical regions. The former classification is possible by the implementation of sequential algorithms through a hidden Markov model (HMM) formulation, and the latter through the use of a combination of cleverly designed sequential detection algorithms. We envision an arsenal of algorithms of this type to be developed in the future.

Video of classification results (low resolution). 8 scans ~ 21 million points. Park Avenue (68th to 70th street). Red: vegetation, green: horizontal, blue: vertical. For high resolution click here.

Video of region-growing results (low resolution). 8 scans ~ 21 million points. Park Avenue (68th to 70th street). Red: vegetation, other colors: connected components. For high resolution click here.
 


Real Time Detection of Repeated Structures in Point Clouds of Urban Scenes


Laser range scanners provide rich 3D representations of urban scenes. These scenes are replete with repetitive architectural features such as windows, balconies, and cornices. Processing of dense 3D images is often computationally intensive and occurs offline after acquisition. Here, however, we present an online algorithm for the detection of repetitive features in 3D range scans. Our algorithm creates a function from each scanline by deriving a local measure at each point. Computing the Fourier transform of that function reveals the periodicity of the scene. This robust algorithm forms the basis for novel methods of feature extraction, compression, and registration. What is more this whole process can be executed on-the-fly and integrated into hardware transforming laser scanners into architecture aware devices.

Online Facade Reconstruction from Dominant Frequencies






We present an online method for filling holes in point clouds by exploiting the regularity of urban areas. Sweeping a plane across the scene we compute periodicity, major planes, and occlusions. Extending rays from the laser that have been occluded gives a planar approximation for holes in facades. The periodicity of the architecture is used to vastly improve this approximation yielding facades that seem complete and natural. Both abstract and high resolution mesh data is constructed from the improved point clouds. All this processing is performed online allowing for seamless integration with scanner hardware.


Offline Algorithms For Repetition Detection in 3D



Detection of Windows


In this paper, we describe novel algorithms for the detection of windows, which are ubiquitous in urban areas. Detecting isolated windows is a challenging problem due to the inability of the laser range sensors to acquire any data on transparent surfaces and due to the wide variability of window features. Our approach is based on the assumption that the elements (windows) are arranged in multiple unknown periodic structures making our system robust to single window detection errors. This kind of detection is essential for high-level recognition algorithms, compression methods, registration, as well as realistic visualizations.