1. Introduction

Various kinds of mobile robots have been developed and used in different applications [1-5]. Although some research is devoted to the automatic control of mobile robots [1], most practical robots are still operated by human operators with the help of video images acquired by cameras mounted on the robots [2-5]. Humanoid robots are designed to have joint structures like those of humans, so they can mimic human body movements and perform various tasks that may be dangerous for humans. However, to control these robots successfully, accurate pose data for manipulating each joint must be generated and processed with a very short latency period.

Various methods have been proposed to control different kinds of humanoid robots by estimating the poses of human operators in remote sites [6-12]. Two main approaches are the use of dedicated positioning controllers and the estimation of the operator’s pose from videos. A master arm was used as a dedicated controller to manipulate a slave arm [6]. Both the arms were designed with the same specifications to move in the same way. A slave arm at a remote location was controlled using a signal generated from electromyograph (EMG) sensors attached to an operator’s arm [7]. The EMG sensors measure the electrical activity of the operator’s muscles and generate analog output signals that are fed to a micro-controller to control the robot. This approach has been proven to provide more accurate data than other methods, but it requires multiple special sensors to be attached to the operator’s body before the robot is manipulated.

Another constraint to this approach is anthropometrical differences: when the lengths of the operator’s arm are different from those of the slave arm, it becomes difficult to control the robot. If the dedicated controller provides only the positions of a hand-like end-effector, inverse kinematics (IK)-based solutions can yield a different robotic pose from the human pose. This pose inconsistency can result in the robot colliding with its surroundings and being damaged.

Recently, virtual reality (VR) has been used for teleoperation systems to provide operators with immersive visual feedback about a robot and its environment [11,12]. However, such systems suffer from poor maneuverability without any haptic feedback or physical constructions. Vision-based human pose estimation has been investigated extensively in the field of computer vision and image processing [8-10]. Important features are extracted from an image that can include people. The features are analyzed to generate the corresponding poses, which can be used in many applications, including robot manipulation [13,14] and the creation of animated movies and games [10].

Robot manipulation using human pose estimation algorithms does not require operators to attach sensors or other electronic devices to their bodies. The estimated joint locations are usually represented as 2-D positions within the image, but recently, 3-D pose estimation has been researched [9, 13, 14]. 2-D joint positions have been initially obtained from images, and then 3-D joint locations were inferred based on the human anthropometrical information [9]. Another method [13] involves the measurement of a person's pose using a RGB-D sensor. The data are analyzed with a depth map, and the operator’s pose is numerically calculated and used to control a humanoid robot. 2-D joint positions were extracted from the RGB image using the OpenPose algorithm [10] and coupled with corresponding depth information to estimate a 3-D pose [14].

However, the 3-D pose obtained using such methods cannot be applied directly to robot manipulation due to its lack of accuracy. Refining inaccurate poses requires a huge amount of computation and results in very slow operation (about 5 seconds) [13]. The accuracy of the robotic control is usually considered a top priority at the expense of manipulating speed, so the inaccurate poses are corrected by an anthropometric difference mapping model called HUMROB.

Both the pose inconsistency and the slow manipulating speed prevent humanoid teleoperation from being widely used. Therefore, improvements in both aspects are necessary. We propose a humanoid teleoperation system to manipulate both of a robot’s arms, as shown in Fig. 1. The 3-D pose data for actuating the joint motors of the robot are generated using a vision-based 3-D human pose estimation (HPE) algorithm without dedicated controllers.

For accuracy, two poses obtained from stereoscopic images are fused and refined using simple but effective multiple filtering. Some joints in a pose can be incorrectly located, but more appropriate positions for the joints can be estimated from another pose extracted from a slightly different point of view.

The rest of the paper is organized as follows. In the following section, related works are presented. In Section 3, the proposed teleoperation system is explained in detail. In Section 4, experimental results are presented. Finally, conclusions are presented in Section 5.

Fig. 1. Overview of the proposed humanoid teleopera-tion system. The humanoid robot is naturally manipulated by visually estimating the 3-D pose of the operator. An accurate pose can be determined by fusing two poses estimated from stereoscopic images.

2. Related Work

3. The Proposed Humanoid Teleoperation System

A control system estimates an operator’s pose and sends data to operate a humanoid robot at a remote site using the Robot Operating System (ROS) [17]. Fig. 2 illustrates the process of the proposed teleoperation system, which uses a conventional 3-D HPE method [18]. The proposed system consists of three threads for both HPE of stereo images and for pose refinement. Notably, the threads for both HPE run asynchronously at around 15 Hz, while the thread for pose refinement runs the hybrid filtering in regular cycles at 60 Hz. By doing this, any new pose determined from HPE is used immediately in the pose refinement without a delay. As a result, the effective rate of the pose update is slightly higher than each HPE rate.

In the projective image formation, pose ambiguity can happen, which leads to inaccurate pose estimation. Therefore, two versions of the operator’s pose are obtained from a pair of stereo images in the proposed control system. Some joints in a pose can be incorrectly located, but more appropriate positions for the joints can be estimated from another pose extracted from a different point of view.

A cascade of simple yet effective multiple filters fuses the two poses and refines the pose reliably and stably in real time. In the teleoperation-monitoring system, the determined pose is converted to joint angle data to drive the motors of the robot. Finally, using ROS-Serial, the joint angle data are transformed to a ROS message that can be sent to the humanoid robot over a network.

Fig. 2. Flow chart of the proposed teleoperation system. The three threads indicated in dashed boxes run asynchronously.

3.1 Human pose refinement technique control

Two initial 3-D operator poses denoted by $\mathbf{P}_{i}^{s}$, $s\in \{L,R\}$, are obtained using the HPE method [19] with input stereo images acquired at frame index $i$. $\mathrm{P}_{i}^{\mathrm{s}}=\left\{p_{i}^{s,~ j}\right\}$ consists of the 3-D positions of eight joints in the upper body, where $j\in \{neck,top,lwri,lelb,lsho,rwri,relb,rsho\}$ indicates the joint index annotated in Fig. 3(a). Fig. 3(b) shows the corresponding motor positions and the base position in the humanoid robot LIMS. For the arm of the robot, four motors $\phi _{i},i\in \left[1\cdots 4\right]$, can be actuated. $\phi _{1},\phi _{2}$, and $\phi _{3}$ are located on the shoulder, and $\phi _{4}$ actuates the elbow.

Each joint position is fused effectively by selecting an appropriate one among two corresponding candidate positions in the initially obtained poses and the previous position as follows:

(1)

$ \overset{˜}{\mathbf{p}}_{k}^{j}=\mathrm{MED}\left[\overset{˜}{\mathbf{p}}_{k-1}^{j},\mathbf{p}_{i^{L}}^{L,j},\mathbf{p}_{i^{R}}^{R,j}\right], $

where $\mathrm{MED}[\cdot ]$ represents a median filter that chooses a median value for each component of the input positions. The frame indices $i^{L}=\left\{i,i-1\right\}$, $i^{R}=\left\{i,i-1\right\}$ for both poses can be different because of the asynchronous execution of the HPE. At least one of $i^{L}$ and $i^{R}$ is $i$. $\overset{˜}{\mathbf{p}}_{k}^{j}$ is produced in regular cycles at 60 Hz. The learning-based HPE method sometimes produces irrelevant poses due to pose ambiguity. The median filtering technique is applied to remove any outlier, thereby eliminating such noisy poses in the robotic movement.

A Kalman filter is then used for each joint to smooth the movement:

(2)

$ \hat{\mathbf{p}}_{k}^{j}=\text{Kalman}\left(\left\{\overset{˜}{\mathbf{p}}_{\Theta }^{j}\right\}\right), \\ \Theta =\left\{k,k-1,\cdots ,k-K\right\}, $

where $\textit{K}$ is set to 50 in our experiments. Even though the median filter corrects inaccurate poses, the median results may include repeated poses or rapid pose changes due to irregularities in HPE processing time. The Kalman filter alleviates these glitch effect by smoothing the movement.

Fig. 3. Correspondences between the skeletal posture and the humanoid LIMS (a) The pose configuration of eight joints, (b) The motors located in LIMS and the base position indicated as a green dot.

4. Experimental Results

4.1 Experimental Environments

The teleoperation system was run on a PC with an Intel-i9, a TITAN V, and Windows 10. The stereo camera in the teleoperation system acquires images with a resolution of 1280 x 720 at 25 fps. The two initial poses are obtained asynchronously at about 15 Hz.

The humanoid robot was controlled by a PC with an Intel-i7, 32GB of RAM, a TITAN V, and Ubuntu 16.0. The stereo camera connected to the humanoid robot is identical to that in the teleoperation-monitoring system. The stereo images are encoded and streamed to the HMD for the operator through an HTTP server. The humanoid robot LIMS has a positioning repeatability of 0.153 mm, and the joint positions of both of its arms are updated every 12 ms [19].

4.2.1 System Operating Rate

We measured the runtime speed of our system and compared it to a system using the CENTAURO robot [14], as shown in Table 1. The frequency output of the RGB-D sensor’s color image and point cloud is approximately 30~Hz. This drops to 8.07 Hz at the OpenPose node when tracking both the human body and hands, which matches the OpenPose benchmark. Finally, the pose information after refinement is delivered to the robot every 140 ms.

In the proposed method, the HPE module estimates two versions of the operator’s pose from stereo images at about 15 Hz. Notably, the two poses are determined in 3-D rather than the 2-D pose obtained by OpenPose [14]. The rate of the filter-based pose-refinement process in our system is 60 Hz, which is 8.5 times faster than the previous system [14] and improves stability by providing more detailed information about where the robot must reach

Table 1. A summary of the frequency at which the data messages are published from ROS.

Method	Stage
	Frequency (Hz)
[14]	OpenNI2	OpenPose	Pt2Xbot
	29.82	8.07	7.13
Ours	HPE		Filtering
	14.97		60.00

4.2.2 Filtering-based Pose Refinement

Fig. 6 illustrates that the pose fusion module selectively collects appropriate joint positions even with the presence of inaccurate estimation. Fig. 6(a) and (b) show three consecutive poses estimated using HPE with the left and right frames of the stereoscopic video. The poses in Fig. 6(b) show an abrupt change since the pose at $k_{0}+1$ was inaccurately estimated due to pose ambiguity. The median filter produced a pose at $k_{0}+1$, as shown in Fig. 6(c), where the poses indicated by red arrows are the input. The resultant pose was properly corrected through the median filter.

Fig. 7 illustrates the pose smoothing by the Kalman filter. The HPE performed in a separate thread irregularly takes a longer time to estimate a new pose. In this case, the pose fusion module applied at 60 Hz can use identical poses as input several times and produce the same resultant poses as in Fig. 7(a), where an identical pose is repeated from $k_{0}+1$ to $k_{0}+3$. In addition, this situation generates a rapid pose change between $k_{0}$ and $k_{0}+1$. The Kalman filter alleviates the glitch effect by smoothing the movement.

Fig. 4. Pose fusion of Y position of the right shoulder by median filter. In the red box, the two poses obtained from stereo images indicate very different directions. In this case, the previous position can be maintained through the median filter.

Fig. 5. Example of pose refinement with respect to the right wrist’s X position. The Kalman filtering increases the stability of joint motion by smoothing out the rapid change of the fused pose in the green box.

Fig. 6.Pose correction in the pose fusion module. Each subfigure represents three consecutive poses estimated using HPE (a)-(b) Poses estimated using the left images and the right images of the stereoscopic video, respectively, (c) Poses obtained using the pose fusion module of Eq.(1). The inaccurate pose at in (b) was effectively corrected by the pose fusion.

Fig. 7. Pose smoothing by Kalman filtering (a) The resultant poses of the pose fusion, (b) The output of the Kalman filter.

5. Conclusion

We have proposed a robot teleoperation system in which the operator's pose candidates are initially obtained using stereo images, and an accurate pose can be estimated using a simple yet effective cascade filter. Unlike conventional systems that slowly control the robots with a focus on motion accuracy, the proposed system can accurately and remotely manipulate a robot in real time. The experiments confirmed that the proposed system teleoperated the robot in real time efficiently and generated accurate motion compared to the conventional systems.