2.1 PointNet Architecture

Qi et al. ^[4] designed a deep learning framework named PointNet, which uses unordered points directly as input. The PointNet architecture is shown in Fig. 1. PointNet basically has three main components: a local feature extraction layer, a symmetric function for summarizing information from all local features, and a global feature extraction layer for aligning global features for various learning tasks. The point cloud is represented by a set of 3D points $\left\{P_{i}|i=1,\cdots ,n\right\}$, where each point $P_{i}$ is a vector containing $\left(x,\,\,y,\,\,z\right)$ coordinates plus an additional feature channel.

A multilayer perceptron (MLP) extends the original 3D points in the point cloud to very high dimensions. In this way, a local feature of a 3D point can be output. The characteristics of each point are shared in the PointNet model. Based on this sharing, the features of the high-dimensional points become different. Considering all the local features of the point cloud, the values of one dimension are integrated into a set of PointNet local eigenvalues in a local feature dimension. Thus, PointNet can use a symmetric function to select a representative value for each set as an output that includes global features of the point cloud.

This step is implemented using an $n\times 1$ max-pooling operator, where $\textit{n}$ is the number of points in the observation point cloud, and the representative value is the maximum of each individual dimension value set ^[5]. This technique solves the problem of out-of-order points being represented by point clouds. After the global features are extracted, these global features are used by the MLP to achieve different goals, such as object classification and semantic segmentation.

Fig. 1. PointNet architecture.

2.2 PointSIFT Module

The PointSIFT module is implemented based on a SIFT algorithm and involves two key attributes: orientation-encoding and scale-awareness. Fig. 2 shows the architecture of the PointSIFT module. Orientation-encoding (OE) convolution is the basic unit in the PointSIFT block that captures surrounding points. Fig. 3 shows the OE unit of the PointSIFT module. Given a point $p_{0}$, its corresponding feature is represented by $f_{0}$. The 3D space with $p_{0}$ as the center point can be divided into 8 subspaces (octants) in 8 directions.

From the $\textit{k}$ nearest neighbors of $p_{0}$, if there is no point in the search radius $\textit{r}$ within a certain octant, the feature of the subspace is considered to be equal to $f_{0}$. Assuming that the input point cloud is $n\times d$, after the step is over, each feature has information in eight directions around it, which becomes $n\times 8\times d$. For the convolution operation to sense the direction information, a three-phase convolution is performed on the $\textit{X}$, $\textit{Y}$, and $\textit{Z}$ axes. For feature encoding of the searched $\textit{k}$-nearest neighbor points, $M\in R^{2\times 2\times 2\times d}$, the first three dimensions represent the coding of the points on the eight subspaces. The three-phase convolution is expressed as:

where $A_{x},A_{y},\,\,\mathrm{and}\,\,A_{z}$ are the convolution weights to be optimized; $Conv_{x}$, $Conv_{y}$, and $Conv_{z}$ represent the respective convolution operations on the $\textit{X}$, $\textit{Y}$, and $\textit{Z}$ axes; and $g\left(x\right)$ represents $ReLU\left(\textit{BatchNorm}\left(\cdot \right)\right)$.

The scale-awareness of the PointSIFT module is formed by stacking direction coding units. A higher-level OE unit has a larger receptive field than a lower-level OE unit. By constructing a hierarchy of OE units, we obtain a multi-scale representation of local regions in a point cloud. For a certain direction coding convolution unit, the features in the 8 direction fields are extracted, the receptive field can be regarded as the $\textit{k}$ field in 8 directions, and each field corresponds to one feature point.

The features of the various scales are connected by several identification shortcuts and transformed by a point-by-point convolution of another output $\textit{d}$-dimensional multi-scale feature. In the process of joint optimization feature extraction and point-by-point convolution of integrated multi-scale features, the neural network learns to choose or adhere to appropriate scales, which makes our network-scale software possible. Ideally, we stack $\textit{i}$ times, and the receptive field is $8^{i}$ points. Finally, these layers are spliced together through a shortcut followed by pointwise convolution ($1\times 1$ convolution) so that the network can choose the appropriate scale with training.

Fig. 2. Architecture of PointSIFT module.

Fig. 3. Orientation-Encoding unit (a) Point cloud in 3D space, (b) Nearest neighbor search in eight octants.

2.3 S-PointNet Architecture

We designed a new semantic segmentation algorithm for point clouds named S-PointNet, which is based on the PointNet architecture. The PointSIFT module is integrated into the PointNet architecture to improve the representation ability. Fig. 4 shows the S-PointNet architecture. The PointSIFT module is applied to extract the local features. By combining the local features and the global features, we can extract new features for each point and perform semantic segmentation.

The proposed deep learning framework S-PointNet directly uses disordered point clouds as inputs. The input to the algorithm is $\textit{n}$ points with dimension $\textit{d}$ (e.g., $\textit{x, y, z,}$ color parameters, and normal vectors). Each input point is first extended to a vector with 64-dimensional features using the MLP. Then it is connected to the PointSIFT module to perform two 64-dimensional transformations to learn the local orientation of each point and output it.

The entire point cloud is then expanded to 1024 dimensions using 3 dimensionally expanded MLPs, which is sufficient to preserve almost all the point cloud information. The output vector matrix is symmetrically max-pooling and takes on global features. The vector operated by max-pooling retains the global feature and loses the feature of a single point, so we use the reshape function to map the vector with global features to all points and join it to the previously obtained local feature vector. Thus, each point can search among the global features and find the category to which it belongs.

The vector is then reduced to 128 dimensions using 3 MLPs of progressively lower dimensionality. Finally, we output data for $\textit{m}$ categories using the full join layer, where \textit{m} is the number of object categories for all points. We also add $\textit{ReLU}$ and $\textit{BatchNorm}$ functions to all MLPs and the full join layer to reduce overfitting.

Fig. 4. S-PointNet architecture. MLP stands for multilayer perceptron, and numbers in bracket are layer sizes.