1. Introduction

UAV use is growing in areas such as scientific research, rescue missions, commerce, and agriculture. Originally, UAVs were developed to be managed by an on-the-ground pilot via remote-control communication ^[1]. Recently, UAVs have been moving closer to navigating with unusual degrees of autonomy. Most UAVs employ global navigation satellite system technology and inertial sensors to determine their geospatial positioning. It is necessary to overcome factors such as GPS signal error, narrow passageways, and transparent glass for stable-flight UAVs in indoor environments ^[2]. Studies in image-based stair-recognition for robots ^[3] and of techniques for ground robots ^[4] are ongoing; however, there is a lack of such research with UAVs. An abundance of techniques, varying from learning-based to non-learning-based, have been suggested to resolve UAV navigation dilemmas. The most popular non-learning-based method is sensing and avoidance, which prevents accidents by steering vehicles in a reverse orientation and navigating by path planning ^[5,6]. Another type of non-learning-based technique takes advantage of simultaneous localization and mapping (SLAM). The inspiration is that, after creating a map of the surroundings by utilizing SLAM, navigation is accomplished by path planning ^[7,8]. The work in ^[7] combines GraphSLAM ^[9] with an online path planning module in a proposal-approving UAV to determine obstacle-free trajectories in foliage. A general characteristic of non-learning-based approaches is that they demand precise path planning, which may result in unanticipated failures when environments are extremely dynamic and complicated. To address this matter, machine learning (ML) methods such as imitation learning and reinforcement learning (RL) have been explored ^[10-^12]. For example, a model-based RL approach called TEXPLORE ^[12] was presented, which is a high-level control system for navigation of a UAV within a grid map having no barriers. And an imitation learning-based controller utilizing a small set of human displays was presented that obtains reliable performance in forested areas ^[10].

Therefore, this paper proposes a convolutional neural network (CNN)-based system based on real-time stair recognition that can fly a UAV without colliding with stairs, and that obtains distance information between walls or stairs through 2D light detection and ranging (LiDAR) with a camera mounted on the UAV. In addition, algorithms were designed for systems that recognize stairs, avoid collisions, and maneuver themselves, which is one of the obstacles to an autonomous flight process, and flight experiments were carried out after the actual UAV was implemented.

Deep learning (DL), which is a subcategory of machine learning, acts like the human brain, and is therefore known as artificial intelligence (AI). Many applications of machine learning have been proposed, with different signals representing data such as music signals ^[13], 2D signals or images ^[14], and video signals ^[15]. CNNs are used for various purposes, such as classification, detection, and pattern recognition, especially in health ^[16], drone applications ^[17], and autonomous driving systems. Recently, You Only Look Once (YOLO) was introduced for real-time detection of objects, with each version improving the mean average precision (mAP) per frame per second ^[18].

In this work, we attempted for the first time to use the YOLOv3-tiny model, and improved the model further by adding a convolution layer to extract deep features for the detection of stairs. This DL detection model was used in a classification problem to determine each next maneuver.

The rest of this paper is organized as follows. Section 2 details related work, while Section 3 explains the proposed scheme. Section 4 summarizes the experimental results and the analysis. Section 5 provides concluding statements and suggests the scope of future work.

2. Related Work

Previously, a 3D map of the local area was developed for autonomous UAV navigation. In some cases, these methods were used to map exact quadcopters ^[19,20]. However, these methods are based on a smart control scheme, thereby restricting their use to laboratory settings ^[21-^23]. The map is learned through other manual route methods, and quadcopters travel the same path ^[24]. For most outdoor flights (where precision is not as high as indoors), a GPS-based posing projection is used.

Most applications use scale sensors, such as infrared sensors, RGB-D (red, green, blue depth) sensors, or laser range sensors ^[25]. A single ultrasonic sensor was used in ^[26] as an automated navigation device with an infrared sensor. The condition evaluation method of the LiDAR and inertial measurement unit (IMU) was advanced to work independently in uncertain conditions that are denied by a GPS ^[27]. Range sensors have limitations, being heavy and high in power consumption.

The simultaneous localization and mapping (SLAM) technique uses separate optical sensors to create a 3D image ^[21-^23] from every UAV position on the map. A 3D map of an unknown indoor scenario was used for the SLAM laser range finder ^[25]. The SLAM technique ^[29,31] offers single-camera indoor navigation. SLAM is highly complicated when it comes to regenerating the 3D map region, requiring precise measurements and extensive resources because additional sensors are needed.

SLAM can also set contact delays during real-time navigation. The studies in ^[31] and ^[32] addressed these issues. SLAM is primarily a practical system, and its output with indoor materials (such as walls/roofs) is not considered good, because its differential intensity is very weak. The entire corridor comprises partitions, roofs, and floors, and SLAM technologies cannot attain the desired navigational quality.

3. The Proposed Scheme

This section discusses the system configuration for UAV recognition of stairs, the deep learning model using YOLOv3-tiny, and the improved YOLOv3-tiny for detecting stairs.

3.1 System Configuration

The proposed system was designed based on recognizing stairs with a camera mounted on the UAV for indoor environments and on distances measured via the 2D LiDAR sensor attached to the UAV’s side. Fig. 1 shows the flowchart for the entire system. The connections and communications between the parts are both wired and wireless, as shown in Fig. 2. In particular, communications among the ground control station, the UAV, and the onboard PC is via Wi-Fi/LTE. Meanwhile, the wired connection is only used for the sensor.

The system’s actual implementation uses a Parrot Bebop 2 drone, which is suitable for narrow passageways and convenient for load sensors. The UAV is equipped with an RPLiDAR S1 laser scanner, which rotates 360$^{\circ}$ and can measure distances up to 40m with a lightweight, mainboard Jetson TX2 embedded computing device (Auvidea J120 carrier board) as shown in Fig. 3(c). The Lenovo ThinkPad T580 is used as a ground control system (GCS), and the equipment required for the experiment is listed in Table 1. All algorithms are implemented in Python, and the Robot Operating System (ROS) was used as middleware (software that can run multiple different programs together) in a kinetic version.

Fig. 1. Flowchart for the proposed implementation.

Fig. 2. Network connections and the architecture of the proposed system.

Fig. 3. System configuration: (a) UAV movement axes; (b) illustration of the RPLiDAR S1 scanning process; (c) the 2D-LiDAR sensor and the Jetson-TX2 onboard PC attached to the UAV; (d) the test environment.

Table 1. Experiment Parameters.

Device	Model name	Company
Lidar sensor	RPLiDAR S1	Slamtec
UAV	Bebop drone 2	Parrot
Onboard PC	Jetson TX2	Nvidia
Carrier board	Auvidea J120	Auvidea
GCS	ThinkPad T580	Lenovo
LTE modem	LTE USB Stick	Huawei

Algorithm 1. Stair-climbing algorithm.

The LiDAR sensor uses distances measured along 360 points, as shown in Fig. 3(b). The distance data obtained by the LiDAR sensor were 0$^{\circ}$ to the floor, 90$^{\circ}$ to the front, and 180$^{\circ}$ to the ceiling, based on the direction of progress for the UAV. In the polar coordination system, each of the raw laser points is defined as $\{(\textit{d}$_${i}$, ${\theta}$$_{i}$); 0 ${\leq}$ $\textit{i}$ ${\leq}$ 359$\}$, where $\textit{d}$$_{i}$ is the distance from the UAV center to the object, and ${\theta}$$_{i}$ is the relative angle of measurement. The information obtained by the LiDAR is stored as a vector $(\textit{d}$$_{i}$, ${\theta}$$_{i}$), and the stored data are checked to convert the values of the infinity scan.

3.3 Deep Learning Model for Detection of Stairs

In this study, a DL approach is implemented for detecting stairs, which the drone uses to make decisions intelligently in order to follow the stairs and determine the next maneuver. In this work, we improved the YOLOv3-tiny default model. The backbone of YOLO is $\textit{darknet}$, where the YOLOv3-tiny default model uses six max-pooling and seven convolution layers. We modified it by adding one more layer. Instead of the softmax function, and where multi-class classification and detection is an issue, regression is employed to solve the multi-class detection and classification problem ^[33].

The proposed model starts by dividing the stair-image input into a G ${\times}$ G grid in the training stage. A bounding box is used as a tool for labeling five features—width $\textit{w}$, height $\textit{h}$, vertical height $\textit{v}$, horizontal height $\textit{u}$—as shown in Fig. 4, and confidence score $\textit{C,}$ which represents the presence of stairs within the bounding box, and hence, represents the accuracy.

Fig. 4. Definition of the bounding box.

Fig. 5. ROS node graph.

Fig. 6. YOLO models: the default YOLOv3-tiny and the improved YOLOv3-tiny.

In the proposed YOLOv3-tiny method, we attempt to make the model computationally inexpensive, along with implementing it to extract more semantic features. Max-pooling is used after each convolution layer to reduce the computational complexity and improve image feature extraction. Fig. 6 shows the network architecture for both the default and the improved YOLOv3-tiny models. The loss function is obtained as an end-to-end network, and can be expressed as follows ^[33]:

(1)

$loss=\sum _{i=0}^{S^{a}}\,\textit{iouErr}+\textit{coordErr}+\textit{clsErr}$

where $\textit{iouErr}$, $\textit{coordErr}$, and $\textit{clsErr}$ indicate the IOU error, coordinates error, and classification error, respectively. We used a rectified linear unit (ReLU) as an activation function to achieve sparsity and reduce vanishing gradient issues ^[25]. Table 2 details the training configuration employed for both YOLOv3-tiny and the proposed improved YOLOv3-tiny model.

Table 2. Training Parameters for Both Models.

Parameters for training	Configuration values
Image/stairs	428 x 428
Batch size	32
Learning rate	0.001
Optimizer	Stochastic gradient descent
Decay	0.0005
Momentum	0.9
Epochs	20,000

4. Experimental Results and Analysis

A dataset was created in the Kumoh National Institute of Technology, South Korea, by employing a Bebop drone that has a high-resolution camera and a GPS mounted on it. The dataset comprises 1000 images at a resolution of $1920\times 1080$ resized to $428\times 428$ before model training. For training and testing purposes, the dataset was split 70% and 30%, respectively. Fig. 7 depicts the training phase of the proposed improved YOLOv3-tiny model where 20,000 epochs were set. As shown in Fig. 7, the blue line represents the average loss achieved (0.215) whereas the red line represents the highest mAP (91.6%).

The detection performance of the improved YOLOv3-tiny model was benchmarked against the default model by utilizing the same parametric configurations and dataset. The metrics used to reflect the efficacy in stair detection of both models are accuracy, recall, F1-score, and precision. Table 3 shows that the proposed improved YOLOv3-tiny model outperformed the default model in terms of accuracy, recall, and F1-score. Furthermore, a low precision value with higher values of other performance metrics shows stable performance from the model.

Fig. 8 shows the real-time detection of the proposed model, where the top left image represents the starting point of the UAV after takeoff, and the top right image represents the middle position of the UAV when hovering and climbing. In Fig. 8, the bottom left image shows the last step of the stairs, while the bottom right image shows the instant when the UAV was located at a distance of $r~ $ meters from the stairs.

For the experimental scenario, the set of stairs climbed was 2.1 m long and 2.85 m wide, as shown in Fig. 3(d). Based on Algorithm 1, Some of the experiment’s results are shown in Fig. 9, depicting commands sent by the GCS and the corresponding images from the built-in camera of the UAV. In Fig. 9, we have tried to show the different stages in the decisions made by the UAV, such as moving forward or upward, hovering, and going to the next stair to climb it. Furthermore, the actual trajectory-wise UAV movement from the beginning of the staircase to the beginning of the next step is shown in Fig. 10 as a 3D plot. This movement started at approximately 0.8 m from the starting point of the stairs. In total, 88 experiments were performed three times each, and the results are shown in Table 4 for the time elapsed during takeoff and landing on average, reported to be 55.97 sec.

Fig. 7. Training phase of the improved YOLOv3-tiny.

Fig. 8. Detection results from the improved YOLOv3-tiny model.

Fig. 9. GCS screen commands and screen shots from the UAV’s built-in camera for (a) forward movement; (b) upward movement; (c) hovering; (d) going to the next stair.

Fig. 10. Trajectory of the UAV.

5. Conclusion

In this study, we designed, implemented, and experimented with a system in which a UAV recognizes and climbs stairs, which are obstacles often encountered during indoor flight. The system was implemented through a CNN-based imaging process for real-time stair recognition and by using LiDAR-based distance measurements. The accuracy derived from stair recognition was 92.06%, and the actual test results showed that stair climbing was carried out without collisions.

Future research would require more efficient algorithms to climb various types of stairs. Moreover, the proposed system can be combined with SLAM navigation to expand studies to systems that can autonomously fly through multiple floors.