Controlling the motion of a two-arm mobile robot through virtual reality techniques

dr inż. Jarosław Jankowski (jajan@ciop.pl) – Centralny Instytut Ochrony Pracy – Państwowy Instytut Badawczy W wielu sytuacjach korzystne może być zastąpienie wykonującego pracę człowieka mobilnym robotem. Dotyczy to zwłaszcza przypadków związanych z przebywaniem w strefie zagrożenia i narażeniem na szkodliwe substancje chemiczne lub promieniowanie jonizujące. Niestety rozwój technik sztucznej inteligencji nie pozwala jeszcze na budowanie w pełni autonomicznych robotów. W związku z tym uzasadnione jest wykorzystanie dwuramiennych robotów mobilnych, pełniących rolę awatara sterujących nimi człowieka. Jako interfejs sterowania dla teleoperatora robota można z powodzeniem zastosować aparaturę typową dla interaktywnych środowisk wirtualnych. SŁOWA KLUCZOWE: roboty mobilne, rzeczywistość wirtualna, teleobecność, zdalne sterowanie

Understanding the essence of individual elements of building the impression of telepresence is not possible without the knowledge of teleoperation technology applications.The most frequently mentioned in the literature include: space exploration, industry, medicine, the army, non-safe professions, as well as education or the entertainment industry.For a more complete understanding of the tele-operation issue, it is necessary to define and analyze its various elements.

Control architecture
This is the way in which a local teleoperation station communicates with a remote location.The basic problem here is the informative feedback between the operator and the device, because the method and quality of the control exercised by the operator via the system depends on this.Factors such as delay or precision of effectors must be taken into account at the design stage of the system in order to create real-time communication and ensure correct performance of tasks.Naturally, the operator's reaction time also affects communication.Research shows that this time in a simple task of pressing a key in response to a stimulus is on average 215 ms [1].However, it is vain to look for simple tasks in the case of teleoperation.The response time of the operator to the complex stimulus in the case of performing or supervising a multistage task can be extended to several or even several seconds (depending on the task).
Teleoperation is usually necessary and used in situations where the distance between the operator and the device (manipulator) is very large.These are usually tasks in an environment that poses danger to people, for example in the depths of the ocean, space or nuclear power plant.This means that there is no escape from delay in the operatormanipulator loop.

Haptic feedback
Haptics refers to the sense of touch, and in particular to the reception of tactile sensations and manipulation of objects, the use of the sense senses and proprioception in order to perform the task correctly.If the haptic feedback is abnormal, then there is a lack of, for example, the impression of resistance posed by objects and the force necessary to move them.This results in a longer task execution time [2] and an increase in the pressure force in comparison with direct manipulation [3].Returning information to the operator about these forces enables the task to be carried out correctly [4].

Human-machine interface
Its function is to provide the operator with information about the remote environment (in which the manipulator is located) to enable proper and comfortable performing of the task.The interface exchanges information from sensors of a remote device into signals that are later picked up by the operator's senses.It is only possible to increase the impression of telepresence if the operator uses a convenient and ergonomic interface.

Description of the test procedure
The purpose of the planned research is to assess the impact of system features such as feedback delay, the level of the telepresence impression and the complexity of the task on the cognitive and physiological functioning of the operator, as well as on the level of task performance.For this purpose, the research method proposed by Yang and Dorneich will be adapted [5].Changes in the research environment will be introduced by changing the complexity of the task and time pressure.Two levels of environmental complexity will be adopted, in which the robot operates remotely (fig.1).The task of the surveyed persons will be to find and collect 14 objects arranged in labyrinths.Regarding the time pressure, two levels are expected: time constraint or lack thereof.The time limit will be determined on the basis of pilot trials.
During the study, it is planned to collect data on such variables as:  perceived task difficultychecking the effectiveness of manipulation,  emotional state (the scale of emotions and the depth of NASA TLX frustration),

Simple environment
Complex environment The tests will be conducted in two rooms (fig.2).In the first one, with an area of 154 m 2 (11 m × 14 m), a volunteer will move.His motion will be recorded using a video motion capture system (28 Qqusysis Oqus 7+ cameras) and inertial methods (to measure the movement: headthrough electronics built into VR goggles, handsthrough wireless gloves of virtual reality).Remote controlled robot (his design, described in detail in [6], is shown in fig.3), whose movement will be representative of the movement of the teleoperator (volunteer), will be placed in the next room with a larger area.

Data transmission
As part of the system, data transmission must be carried out in two ways.On the one hand, data must be sent continuously to control the robot's movement (fig.4), and on the other handthe teleoperator must be constantly informed about what is happening in the robot's environment (this is achieved by real-time image presentation from a pair of cameras)on the robotfig.5).Recorded data on the movement of the teleoperator relate to the position and orientation of the hand, head and torso in three-dimensional space, as well as the degree of flexion of the fingers.Location data is recorded by the video motion capture system, while the orientation databy AHRS electronic systems mounted on the body, gloves and virtual reality goggles type HMD.The degree of flexion of the fingers is recorded by the sleeve communicating with a PC computer via a Wi-Fi wireless network.These data are then integrated and sent to the computer that controls the movement of the robot components.There they are converted into control signals for specific motors in such a way as to represent the motion of the teleoperator.The movement of the wheeled platform will reproduce the trajectory of the teleoperator's trunk with regard to its orientation, which means that the robot will rotate in a place where the teleoperator will rotate around its own axis.The head's rotation relative to the trunk will be reproduced by motors controlling the movement of both cameras installed on the robot.It is planned to apply two degrees of freedom in terms of camera rotation.The head movement in the vertical axis will be reproduced by means of a line module, on which a pair of cameras together with engines that reproduce the head rotation are installed.If the operator for some reason lowers his head (e.g., squats), the line module will lower the position of the cameras so that they are at the same height as the HMD image.Similarly, the movement of the hands will be mapped by the movement of the linear modules on which the arms are fixed, and the motors causing revolutions around the axes corresponding to the individual joints of the upper limb (shoulder, elbow, wrist).The position of the robot gripper relative to the pair of cameras will be positioned as close as possible to the location of the teleoperator's teleoperator, relative to the HMD virtual reality goggles.This task is relatively simple, because the offset in the vertical axis does not depend on the movement in the remaining dimensions, i.e. the height at which the robot arm is located is directly dependent on the height on which the virtual reality glove is located.Thanks to this solution, the reverse task is much easier.The gripper clamp will be adjusted by the degree of flexion of the fingers, recorded by the virtual reality gloves.It is planned to protect the gripper motors against potential damage by monitoring the pressure force.
A separate problem is the transmission of video data from a pair of cameras installed on the robot to a teleoperator.A stereoscopic image will be sent, enabling the perception of depth and proper determination of spatial relations between objects, including between the robot gripper and the object to be transferred.These data should be sent with the least possible delays.Images from cameras will be sent to a computer, whose task will be to combine them in a way that ensures their display in virtual reality goggles.The use of cameras with a connector responsible for synchronizing their work allows you to register both images at the same time.Otherwise, the images could be different, which would cause discomfort to the teleoperator.The images will then be compressed and sent over the air to be displayed using virtual reality goggles.

Conclusions
The applications of combined virtual reality technology and teleoperation have been reviewed.It has been shown that teleoperation can benefit from the inclusion and improvement of virtual reality techniques.The issue of telepresence is the key here.Improving the virtual reality technology through the application of research results on increasing the immersiveness of virtual environments will translate into better functioning of teleoperation systems.This will be done by increasing the impression of the operator's presence in a distant environment due to the VR interface.Stereoscopic vision, freedom of movement, natural reflection of movements, real-time feedback, including haptic feedback, increase the impression of telepresence, and thus allow more precise performance of entrusted tasks.The future of teleoperation lies precisely in the integration with virtual reality technology.
In summary, the applications analyzed are:  Industry.The main advantages of teleoperation and telepresence in industry are the decreasing costs associated with maintaining and increasing productivity and reliability by replacing people with robots, and thus higher incomes to guarantee return. Medicine.Remotely operated robots can perform surgical operations in remote locationssuch as a battlefield or space stationor on a cellular scale.In addition, thanks to virtual reality, the prevention and treatment of diseases can be achieved using commonly used VR-based devices. Education.Teaching methods change thanks to technological progress, which allows to achieve better learning outcomes.
 Drones.The number of their potential applications is constantly increasing, such as avoiding dangers at work or performing tasks automatically.Further applications will be developed to increase safety and comfort of life. Hostile environments and unsafe workplaces.Thanks to teleoperation, telepresence and virtual reality, progress has been made in the exploration of the cosmos and depth of the oceans, and the effectiveness of actions in the case of natural disasters, fires or other emergencies is increasing.The special role of virtual reality, however, consists in a radical change in the lives of people with disabilities, e.g.due to the rehabilitation of motor disorders [7] or the support of physical work through remotely controlled robots.

Fig. 1 .Fig. 2 .
Fig. 1.Exemplary implementation of changing the difficulty of the task by forcing the implementation of more complicated trajectories during the process of reference in the laboratory

Fig. 4 .
Fig. 4. Diagram of data transmission to the robot.Data between labs will be sent via Ethernet

Fig. 5 .
Fig. 5. Diagram of data transmission to the teleoperator based on the results of the fourth stage of the multi-annual program "Improving safety and working conditions", financed in 2017-2010, in the field of scientific research, by the Ministry of Science and Higher Education.Program coordinator: Central Institute for Labor Protection -National Research Institute.