A hands-on approach to distance medicine

Dr. Pijuan Yu sits at his lab station, focused on the task at hand. He is feeling for a lump. The glass wall to his right reveals other students and researchers as they pass, but his attention remains steady.

Two rubbery samples simulating human neck tissue lie on a smooth black tabletop. A dark cloth covers the first, preventing Yu from seeing clues in the material.

Yu takes his time considering what he feels. There’s a method to examining the phantoms, as he calls them — center, left, right, up, down, back to center. He evaluates both samples before correctly identifying which has the lump. 

Simple enough. 

Except Yu never touched either sample. He wasn’t even in the same room. 

Yu flashes a smile down at his phone, where his labmate, postdoctoral researcher Dr. Anzu Kawazoe, is on FaceTime across the hall, physically performing the exam. 

“Let’s set up another,” Yu says.

Bringing touch to telemedicine

Telemedicine commonly uses video calls to reach patients near and far, but the technology limits what senses caregivers can use during their evaluation. 

Yu and his collaborators at Texas A&M University are seeking to go beyond telemedicine’s use of audio and video. They want to add an entirely new dimension: touch.

In 2023, the National Science Foundation (NSF) awarded researchers from the College of Engineering a $1.5 million grant to develop haptic technologies to advance telemedicine. Dr. Rebecca Friesen, assistant professor of mechanical engineering, leads the project to develop technology that can digitally transfer touch just as audio and video are conveyed now.

A range of consumer technologies already use haptics — think of the rumble of a video game controller, responding to an on-screen action, or the vibration of a steering wheel when a car drifts out of its lane. The word comes from the Greek haptikos (meaning “relating to the sense of touch”) and often refers to technologies that create physical sensation for users.

Friesen, who has been Yu’s Ph.D. advisor throughout his four years at Texas A&M, said that her lab is just one part of a much broader effort. 

Collaborators on the project are helping the sensors collect better data, exploring how users interact with the technology, and gathering feedback from medical experts. 

The ability to feel — or palpate — is an important tool caregivers use when evaluating patients. If successful, the technology researchers are developing could be used in clinical settings, allowing a nurse or technician to visit a home-bound or rural patient, perform a palpation exam, and send the video and haptic recording to a physician — located in another city, state or country — for interpretation. The haptic experience, along with a video call to the patient, would aid the physician in making their diagnosis.      

The researchers envision the following: A nurse, located in a rural clinic, performs a palpation exam while wearing a sensing glove. A camera records the nurse’s hands performing the exam. With just a few seconds of delay, a physician at a medical center — like Houston Methodist Hospital — watches the video as a finger-sized haptic “display” surface simultaneously recreates the sensations captured by the glove. The physician keeps their hand still as the haptic display shifts against it, replicating the pressure applied by the nurse during the palpation. The near-real-time interaction allows the physician, nurse and patient to communicate throughout the exam, enriching the doctor’s assessment and participation.

The haptic display uses a cluster of actuators that translate the signals collected by the glove’s sensors into tactile motion. Effectively, the haptic surface is a tiny robot, shapeshifting against the user’s finger to mimic the information the sensing glove collects. 

The team’s ultimate goal is to make the technology disappear from the user’s perception, tricking the physician’s senses into thinking they are interacting with a real patient. When the motion of the haptic display syncs with the exam’s video, the user is more immersed in the experience, making the hands performing the exam feel more like the user’s own.

“When we get the haptic feedback synchronized with the video, it’s really compelling,” Friesen said. “We’ve had some moments of that in controlled settings, and it’s so cool when it works. Those glimpses are very encouraging and very exciting.”

Dreaming of robots

Yu grew up in China, enthralled by the robotics in American cinema — like Transformers and Iron Man — and was determined to help make them real. He began his undergraduate studies in China and then transferred to the Florida Institute of Technology, where he graduated with a bachelor’s degree in mechanical engineering in 2020. 

He continued his mechanical engineering studies as a master’s student at Northwestern University, where his work on a class research project caught the attention of Dr. Matthew Elwin, who invited Yu to work in his lab. The experience taught Yu about robotic controls — the systems that command a robot’s precise movements or tasks — and expanded his understanding of academic research. It gave him confidence in his abilities and encouraged him to apply to Ph.D. programs around the country. 

When Yu received his master’s degree in December 2021, he had planned to stay on as a research engineer in Elwin’s lab as he awaited the results of his Ph.D. program applications. 

However, the lab’s project was out of funding, and Elwin was unable to hire him. 

Before he could resume his studies, Yu needed to find a job. 

A vision of what’s possible

In January 2022, as Yu was searching for work after he graduated from Northwestern, Friesen accepted a faculty position in the J. Mike Walker ’66 Department of Mechanical Engineering at Texas A&M. Her research interests focused on human-machine interaction and the development of haptic technologies. 

While developing the haptic display system over the past few years, Friesen explained that regular feedback from physicians with hands-on experience has been critical to the technology’s progress. 

That feedback comes from the team’s collaborators — including physicians and students — at Texas A&M’s Engineering Medicine (EnMed) program, a four-year program resulting in a medical doctorate from the Naresh K. Vashisht College of Medicine and a Master of Engineering degree from the College of Engineering.

The team holds workshops with EnMed twice a year, either traveling to the Engineering Medicine campus in Houston or hosting the physicians in College Station, to demonstrate their progress and test the machinery. This regular feedback helps ensure that the engineering team builds a device ready for clinical use, rather than a technically impressive technology with little practical application. 

Dr. Ericka Greene, an associate dean of the engineering medicine campus and a physician at Houston Methodist Hospital, is one of the team’s collaborators. She said the technology could lead to portable, user-friendly healthcare devices for caregivers ranging from medical students to nurse practitioners and expert physicians. 

“I’m in awe of where we’re going with this and the direction that biotechnology is taking healthcare,” Greene said. “It’s still at its basic level, but it does give us a vision of what’s possible.” 

As the team refines the technology, Greene is using her years of experience evaluating patients to connect the mechanical sensations to reality. No matter how good the technology becomes, she said, teaching physicians to translate between the mechanical feedback and the sensation of a hands-on evaluation will be important to the project’s success.

“The data I get from feeling a nodule — a mass — on someone’s neck, for instance, I know what that feels like to my fingers after years of doing it,” Greene said. “I know what to expect and look for from the examination, and I don’t have to think about it. Going through this process with our co-investigators, it requires an additional cognitive step to extrapolate to a new sensation.”

Adapting the technology to the human experience is one of the project’s main goals, Yu said. The haptic applications are meant to assist people, not replace them. 

Yu didn’t always view technology this way. But an unexpected opportunity opened his eyes to how engineering could change a person’s life.

From robotics to haptics

After receiving his master’s from Northwestern, Yu spent the first few months of 2022 as an engineering intern, working on control systems for autonomous vehicles at a Chicago-area startup. Then, another opportunity arose. 

Elwin was collaborating with Dr. Netta Gurari of the Department of Physical Therapy and Human Movement Sciences at the Northwestern Feinberg School of Medicine to develop a device to help stroke patients. When Gurari needed to hire a research technician to join her team, Elwin recommended Yu for the role. 

Although he was excited about the opportunity to join Gurari’s lab, Yu remembers being anxious about the adjustment. He was no longer focused on robotic arms and autonomous vehicles; he was dealing with humans. 

“It was a whole new world for me,” Yu said. “I was asked to build a haptic device to help stroke patients, with no background in biomedical science or neuroscience.”

Fortunately, his experience in robotics translated well to his new focus on haptics. 

Yu explained that actuators (responsible for creating motion) and sensors (responsible for detecting changes) are fundamental in both robotics and haptics. In robotics, mechanical sensors drive the actuators. But in haptics, the human body — including the skin and the nervous system itself — functions as both a crucial part of the sensor and the final receiver. 

“Building a haptic device draws on the same engineering knowledge and the same tools as building a robotic device,” Yu said. “The real difference was the focus: everything now centered on the human, and the goal was tied to human perception rather than the machine itself.”

This distinction adds complexity to the development of haptic devices: In addition to the device’s technical function, researchers must account for the user’s experience, perception and participation. Yu needed to consider how the device’s user would interpret the mechanical feedback. 

For example, is the stimulus clear to the user? Does interacting with the device cause discomfort? Do different users experience the stimulus in the same way? For Yu, learning about the technology brought humans into the mechanical world. 

Working in Gurari’s lab let Yu connect his engineering work to outcomes that could affect lives. Starting in his first weeks on the project, he interacted with stroke patients who visited the lab as research participants. 

“That was the first time I realized how technology can help people,” Yu said. “I met with stroke patients and talked with them — they were old and couldn’t feel anything from their hands. It made me realize how important the sense of touch is. I think that experience changed my life.”

Arriving at Texas A&M

In April 2022, just a few weeks after joining Gurari’s lab, Yu received his acceptance to the mechanical engineering doctoral program at Texas A&M. 

As luck would have it, Elwin had a connection in College Station, a former Northwestern Ph.D. student who had recently joined the department: Dr. Rebecca Friesen. With Elwin and Gurari’s endorsement, Friesen welcomed Yu to her lab and became his advisor. 

Yu remembers his anxiety when he arrived at Texas A&M in fall 2022. Although his time in Gurari’s lab had introduced him to haptics, that experience had been relatively short and limited. 

Friesen saw it differently, though. As a new faculty member building her research lab, she needed someone experienced to begin developing new devices immediately. 

She viewed Yu as a perfect fit. 

“My first impression of Pijuan was that when he had a defined problem in front of him, he was going to approach it single-mindedly with 100% of his energy until it was solved,” Friesen said. 

Friesen’s trust and Yu’s work ethic paid off. He accelerated development of the haptic display system in his first few months in the lab, enabling the team to collect pilot data that ultimately helped secure the $1.5 million NSF grant.

Yu said Friesen’s encouragement motivated him throughout his time at Texas A&M and gave him the confidence he needed to flourish on the project. 

“Dr. Friesen’s words motivated me through my entire Ph.D.,” Yu said. “Even now, I wouldn’t call myself an expert because haptics is a huge field. But I’m confident that I know my own research direction deeply.”

Yu defended his dissertation in June and, as he graduates in August, is planning to pursue his original career path in robotics. He is returning to the field with a new perspective on human touch and the importance of reliable haptic systems — two areas he said most humanoid robot projects will need in the next phase of development. 

Specifically, he wants to develop dexterous robotic hands. While current robots are adept in major movements like walking, running and dancing, Yu said the machines’ fine motor skills are lacking. 

“Ask one to manipulate a tool or shuffle a deck of cards, and most of them struggle, because they rely heavily on vision rather than touch,” Yu said. 

It’s a problem Yu has seen before while working in Gurari’s lab at Northwestern. Much like modern robots, the stroke patients Yu interacted with had good vision, but struggled to pick up objects because of poor communication between the sensations in their skin and their nervous system. 

Yu believes humanoid robots face a related limitation.

“If they don’t have rich tactile sensing, they will never reach human-level dexterity,” Yu said.  

By combining his experience with stroke patients and his expertise in haptics at Texas A&M, Yu feels equipped to address the challenge.

What the future holds

Yu sits at a conference table, facing the glass wall of his lab, a short walk from where he demonstrated the project’s progress minutes earlier. From his seat, the prototype haptic display is visible. 

The project has made great strides during his four years working with Friesen and the team. But there is much left to do. 

Yu’s work has shown that remote users can interpret the machine’s haptic feedback, but now he considers where the project will go next. 

In preparation for his departure, Yu has spent the summer documenting the haptic display system’s functions for collaborators, including students in the Human Factors and Cognitive Systems Lab, led by Dr. Thomas Ferris, an associate professor of industrial and systems engineering. Ferris’ group is preparing to examine the human workflow of the device, seeking to improve how nurses, physicians and patients will interact with it. 

But work on the system’s technical development isn’t over. 

Yu’s eyes are fixed on what has been his corner of the lab, considering what new methods or technologies — maybe a new type of actuator — might need to be developed. 

“It’s not easy,” Yu says, grinning. 

In many ways, Yu says, the project faces technical problems similar to those it faced at the beginning. His work on the project addressed the haptic display’s recreation of motion perpendicular to a given surface — known as normal force — but future advancements will need to go further. In real-world palpations, healthcare providers also use lateral movement — shear force to an engineer. If the team hopes to accurately simulate an exam, Yu says, they will need to invent technology to record and recreate this type of motion. 

“It’s a big challenge,” Yu says. “If a Ph.D. student can solve that problem, that’ll create a very big impact.”

Yu’s eyes return to the glass wall. 

Wherever his path leads next, he says he’ll be following the project’s progress closely — from a distance. 

Tags

• College of Engineering
• health care
• mechanical engineering
• research

Media Contact

Alyson Chapman

Texas A&M University College of Engineering

achapman@tamu.edu

Share this News