Virtual reality, videogames and robots as tools in physical rehabilitation

Projects presented during the 4th BRAINN Congress in Brazil combine physical rehab treatments with technology. Fun and electronic games can make sessions more stimulating for patients with motor limitations.

By Karina Toledo, in Campinas  |  Agência FAPESP – Technologies originally designed for entertainment, such as electronic games, gesture recognition sensors and virtual reality (VR) goggles, can make rehabilitation sessions more enjoyable and effective as well as help patients with motor limitations recover at least part of their autonomy. 

Two projects with this aim were presented at the end of March during the 4th BRAINN Congress, held at the University of Campinas (UNICAMP) in São Paulo State by the Brazilian Research Institute for Neuroscience and Neurotechnology (BRAINN), one of the Research, Innovation and Dissemination Centers (RIDCs) funded by FAPESP.

In a pre-conference symposium called “Virtual Reality and Neurofunctional Recovery,” BRAINN researcher Alexandre Brandão presented examples of VR software for motor and neurofunctional rehabilitation. The goal is to make the technology more affordable for stroke patients and others who need medium- to long-term physical rehabilitation treatment.

“Two VR smartphone apps are in the development stage: e-Street, which lets the user virtually explore an urban environment and situations like crossing a street, and e-House, which lets the user walk around the outside of a house to train in fall prevention while virtually going up and down stairs. It will soon be possible to go into and out of rooms inside the house, simulating everyday activities like opening drawers or closets. In addition to constantly developing the application interface, we’re investigating the best way to interact with certain objects in a virtual environment,” said Brandão, who designed the programs and is a postdoctoral fellow at UNICAMP’s Physics Institute, supervised by Professor Gabriela Castellano.

According to Brandão, conventional VR software for smartphones merely takes users on a predefined virtual tour. They can steer by looking in one direction or another but cannot genuinely control the virtual environment. The programs developed at UNICAMP with FAPESP’s support allow users to move through the virtual environment by moving their legs while standing in one spot.

In addition to software, Brandão and his group are developing hardware with the aid of 3D printing. In e-Street, for example, ultrasound sensors worn on the ankles to enable users to walk through a city operate like a sonar, detecting gait cycles and communicating via cables with a small Arduino controller board attached to the user’s waist. The Arduino communicates via Bluetooth with a smartphone inserted into the VR goggles. During e-Street navigation, the direction while exploring the streets of the virtual city is determined by head movements and body rotation.

According to Brandão, the first tests with stroke patients began in March. Although these patients are the main target for the project, and the most challenging, he believes the technology may also benefit older people who suffer from spatial disorientation (frequent in many kinds of cognitive disorders), help prevent falls, and possibly inhibit the pain caused by motor activity.

“Many falls happen when older people have to split their attention between a motor task such as going upstairs or avoiding an object in the street and a cognitive task such as looking into a store window or remembering the way home,” he said. “With this software, it will be possible to train the attention associated with motor stimuli and anticipate situations that could endanger physical integrity.”

Moreover, people with fibromyalgia and individuals of all ages with motor limitations due to fracture, surgery, spinal cord injury or any other type of trauma will benefit from more enjoyable and stimulating physical and occupational therapy sessions despite the repetitive tasks involved.

GestureCollection

However, not all patients are able to cope with a highly immersive virtual environment such as that created by the use of VR goggles and a smartphone, Brandão noted. Some may have headaches and feel dizzy or even experience motion sickness while using the equipment.

In these cases, it is possible to use a different series of apps, called GestureCollection, which Brandão began developing in 2012 during his PhD research at the Federal University of São Carlos (UFSCar) in São Paulo State.

One of these apps is GestureMaps, which lets users navigate from the perspective of Google Street View, walking through many of the world’s cities without leaving home or the rehab clinic. 

Another app, GestureChess, lets the user control a chess game with hand movements. GesturePuzzle is a jigsaw puzzle game that lets the user drag virtual pieces from one side of the screen to the other, and GestureChair is a game in which the player must find a way through a maze and collect balls using arm movements.

Finally, RehabGesture is designed for use by health workers in rehabilitation therapy, such as physical or occupational therapists. The program measures shoulder and elbow movements in real time during interaction with a virtual environment. It records the data for analysis and assessment of a patient’s difficulties and serves as a database for gauging progress.

“GestureCollection programs can be installed on any computer coupled to Kinect-type sensors [used in the Xbox videogame console], which cost about 100 dollars each,” Brandão said.

The patient must be placed in front of the sensor, while the health worker controls the computer and establishes the exercises to be performed and the degree of difficulty. The images can be watched on a monitor or TV screen, or projected onto a wall.

Preliminary tests of the apps are in progress at UNICAMP’s teaching hospital, Hospital de Clinicas, under the supervision of neurologist Li Li Min, and at the Mogi Mirim unit of Rede Lucy Montoro, a rehab center in São Paulo for patients with incapacitating physical, motor and sensorimotor disabilities. Thus far, the software has been used only by specialists to test its usability and determine how it can be improved.

Initial tests in a school environment have also been conducted via collaboration with the degree course in special education at UFSCar and three other departments of the same university: Gerontology, Occupational Therapy, and Physical Therapy. Projects are also being developed at UNICAMP’s Physical Education School (FEF).

Part of the research also benefits from collaboration with Diego Roberto Colombo Dias, currently an assistant professor at the Federal University of São João Del Rei (UFSJ) in Minas Gerais. Patent applications for five of the programs have been filed with INPI, Brazil’s patent office. According to Brandão, INPI granted a patent for e-House in 2016.

Robots and videogames

Also during the 4th BRAINN Congress, Glauco Caurin, a researcher at the University of São Paulo’s São Carlos Engineering School (EESC-USP), presented work on electronic games and robotic members that serve as auxiliary tools for physical therapy. 

Caurin is conducting his research at the Center for Advanced Studies in Rehabilitation (NEAR), coordinated by Linamara Rizzo Battistella, a professor at the University of São Paulo’s Medical School (FM-USP). “The robot acts as a sophisticated joystick,” he said. “The game determines the movements the patient has to make in order to attain a particular goal, which may be picking fruit or preventing a ball from falling off a platform, for example. If the patient can’t perform an action alone, the robot waits a few fractions of a second and then helps.”

The physical therapist or physiatrist responsible for the treatment can modulate the amount of help the robot provides, Caurin explained. The idea is that its assistance should diminish as the treatment progresses. There may be a point at which the robot even starts opposing a movement in order to increase muscle tone.

“Patients have full control over their actions. The robot knows in advance what movement needs to be performed to win the game, but if the patient doesn’t intend to perform this movement, nothing happens. They can lose deliberately if they wish. Our concern is to develop the robot so that it can’t hurt anyone. In one of our lab tests we showed that the machine has sufficient sensitivity to get out of the way if pushed with something fragile, like a potato chip,” Caurin said.

The project being developed at EESC-USP focuses on wrist and ankle rehabilitation, although the group has also created robots to assist shoulder movements and even an exoskeleton capable of assisting lower-body movements.

“The ankle provides about 50% of the energy needed to walk. The ankle plantar flexors contribute up to 50% of the positive mechanical work in a single step to permit forward propulsion,” Caurin said. “So it’s a major achievement if we can recover this movement. Any gain in independence can greatly improve patients’ lives and lighten the burden for family members.”

International studies have demonstrated that robot-assisted physical therapy is more effective than classical rehab, he added, especially for the upper body.

“With a robot, it’s possible to perform almost ten times more movements in a session than with assistance from a physical therapist, as human beings also get tired. Furthermore, videogames make the exercises more stimulating. We’re still working on ways of making the games less static and repetitive, so that patients can advance to the next level,” Caurin said.

Far from losing importance in the rehabilitation process, health workers become supervisors of the entire treatment. To assist them, the group at EESC-USP is developing computer programs that can objectively measure variables such as movement amplitude, acceleration and force. The information is collected while the patient is performing an exercise and stored throughout the treatment to demonstrate the patient’s progress.

Prototype robot members at more advanced stages of development are already being tested in patients through a collaboration with researchers at the University of São Paulo’s Ribeirão Preto Medical School (FMRP-USP).