How do you teach an ER nurse how to palpate the abdomen of a pregnant patient or learn where to position a stethoscope when listening to their lungs if you don’t have an actual patient readily available to practice on?
You create one using augmented/virtual reality and utilizing the expertise and resources of one of the largest research universities in the state of Texas.
At Texas A&M University, Dr. Jinsil Hwaryoung Seo is combining her artistic background with her technical skills to create a safe, immersive learning environment for students. “I create virtual/augmented reality and artificial intelligence applications to support people in challenging situations in the real world,” explains Seo, an associate professor in the College of Performance, Visualization and Fine Arts at Texas A&M and director of the Institute for Applied Creativity.
Through the trainings in those environments, people can experience something not possible in the real world, but they can practice things in the virtual world. Hopefully, we can make an environment for training that can be repeatable and empower individuals.
The institute brings together specialists from a variety of fields to advance traditional education by integrating high-fidelity 3D computer graphics, haptic feedback systems and artificial intelligence.
“Through the trainings in those environments, people can experience something not possible in the real world, but they can practice things in the virtual world. Hopefully, we can make an environment for training that can be repeatable and empower individuals,” she said.
Seo’s work focuses on using immersive technologies to help students master complex tasks and procedures. Instead of memorizing passages from a textbook or watching a video, students can enter a virtual world where they can perform the tasks using virtual reality (VR) headsets and controllers.
Seo says this type of immersive learning can help students learn and retain information better. “Students learn specific concepts through this type of simulation-based experience. By physically interacting with objects in a virtual environment, their bodies engage in embodied learning, which enhances memory retention. As a result, they can more easily apply what they’ve learned to real-world situations,” she said.
Photos by Laura McKenzie/Texas A&M University Division of Marketing & Communications
Real-World Scenarios in the Virtual World
Creating these complex training applications starts with collaborating with faculty and subject matter experts (SMEs) to map out what material the training needs to cover, what the virtual environment should look like and how the student will interact with the module.
“We have forensic nursing training applications, a male version and female version. We have a sepsis program for emergency room training. We created a parent-teen interview, like a conversation practice. We also have a few anatomy education applications. Many of them are being used in classrooms and training situations.”
One of Seo’s early collaborations was with the College of Nursing at Texas A&M. In 2020, she began working closely with faculty from the college to create and launch over 12 different virtual reality programs.
“These VR programs have been instrumental in adding to the body of research for both nursing and visualization on how its use can enhance nursing education in communication, skills and overall clinical judgement,” said Dr. Elizabeth Wells-Beede, a core collaborator at Texas A&M and founding senior associate dean and professor in the College of Nursing at the University of North Texas.
Throughout the development process, Seo and her team meet with the SMEs to make sure the simulations are meeting their needs and course-correcting if needed.
“This process is very iterative. Sometimes we meet with the nursing faculty every week, allowing us to receive feedback from them promptly. And then, once we have a prototype working, we invite them to actually try it,” she said.
“Before we develop a VR application, we can show how it’s going to be shown to the users, and then how the room needs to be designed. So we use that kind of interface to design and then plan for the application with subject matter experts,” Seo said.
Cheng-Yan (Max) Tsai uses haptic gloves, which are equipped with sensors that enable users to experience the sensation of touch within a virtual environment. The gloves can be used in scenarios like assessing the fetal position in pregnant patients.
Once the rough visuals are developed, the SMEs review them with the team and make any final necessary changes. Once approved, a team of visualization students starts work on creating the 3D design.
“The artists will work on finalizing and polishing the environment, and also ensure that all the interaction needs go to the VR or AR [augmented reality] developer so they can put all the interactions into the application. We use programs called Unity and Unreal, so those are the environments we put all together to make it work for VR and AR.”
Seo’s team is composed of a dynamic group of graduate and undergraduate students, including graduate students Dhruv Rathod, who focuses on VR development; Mihir Godbole, a developer specializing in conversational AI systems; and Cheng-Yan (Max) Tsai, who is an Unreal Engine developer currently working on haptic and tangible interaction.
They work alongside Seo and a skilled technical artist staff member, Hayden Harsh, who is a specialist in 3D animation, motion capture and 3D graphics production. This multidisciplinary team is actively involved in all stages of the research process, contributing significantly to both the development and evaluation of cutting-edge interactive technologies.
A vital part of the simulation is the use of virtual patients and other VR characters needed for the training scenarios. Seo’s team uses motion capture technology to bring these characters to life, like virtual patients or doctors, ensuring realistic and immersive interactions within the virtual environment.
“We utilize a complex motion capture system. Then, 3D and animation artists collect motion data from a motion capture suit. After the shoot, the captured data is processed and cleaned up,” Seo said. “Then that motion data is integrated into the application, allowing our virtual characters to move in a highly natural and realistic way.
“We use this technology for creating virtual patients or virtual nurses. So, the trainees and students can practice their conversations and specific scenarios utilizing conversational AI technology.”
The (Virtual) Doctor Will See You Now
This medical VR world creates a safe and controlled training space for students learning techniques that could significantly impact a patient’s health in the real world. But in the VR world, mistakes become valuable learning opportunities — without life-or-death consequences.
“We provide a safe environment for standardized patients as well as students. So students can make mistakes, repeat scenarios as needed, and ultimately master the skills and knowledge through hands-on practice,” Seo said.
For example, for nurses seeking specialized certification through the Texas A&M Health Center of Excellence in Forensic Nursing, an environment has been created to reflect an examination room with a patient avatar. Students interact with the patient, going through each step of the exam as if they are in the room. These steps are guided by the app and monitored by faculty to help them learn trauma-informed, patient-centered care before working with real patients.
“The use of virtual reality has taken forensic nursing education to a new level. We have a new method for skill acquisition that is low stress. Our VR experiences reach our learners wherever they are, anywhere in Texas and beyond,” said Dr. Stacey Mitchell, clinical professor and director of the Texas A&M Health Center of Excellence in Forensic Nursing.
Working with Dr. Seo and her team has been an amazing experience! Our dreams are now a reality. The students under Dr. Seo’s guidance have done incredible work that has real-world impact.
“Working with Dr. Seo and her team has been an amazing experience! Our dreams are now a reality. The students under Dr. Seo’s guidance have done incredible work that has real-world impact,” Mitchell said.
Using VR training can also save money for resource-strapped institutions, especially in rural areas. The cost for running a full gross anatomy lab can be cost-prohibitive for smaller institutions, but virtual reality allows students to access high-quality anatomy training that might otherwise be out of reach.
Dr. Michelle Pine is a clinical professor with Veterinary Integrative Biosciences in the College of Veterinary Medicine and Biomedical Sciences at Texas A&M who worked with Seo’s team during the pandemic on a prototype for a supplemental anatomy learning tool for students.
“We initially focused on osteology (learning the names of the bones and the important structures on each bone). I was the content expert while Dr. Seo and her team created an immersive experience for the students to learn by engaging directly with the content,” Pine said. “The prototype was interactive and gave students the opportunity to identify, sort and compare bones from two species: human and canine. They could even ‘build’ a skeleton.”
Though this was only a prototype, Pine says that the benefit of using VR for anatomy instruction is that it allows students to access and actively engage with the content in a more realistic way without having to be in a laboratory. Seo says that these prototypes have been tested with students from gross anatomy classes, veterinary and medical schools, and have received positive feedback. Based on these promising results, this approach is now being explored for broader application in other educational settings at Texas A&M.
Working Together to Make a Difference
Seo says her work is a truly collaborative endeavor, drawing on not only the expertise of her team but that of the faculty to make sure the trainings meet the needs of the students and provide the best immersive learning experience possible.
“In various learning situations, for example, a sepsis situation in the emergency room, a gross anatomy learning, we collaboratively developed customized VR applications. These applications have been implemented and studied by collaborative faculty with their own students. Our tools have been used in the classrooms, in the nursing school at Texas A&M, as well as the University of North Texas and Texas A&M International University,” Seo said.
Photos by Laura McKenzie/Texas A&M University Division of Marketing & Communications
“Dr. Seo and her students have worked to create state-of-the-art programs that surpass other programs with realism, creativity and attention to detail,” Wells-Beede said. “To date, we have co-delivered presentations at the local, national and international level and published in peer-reviewed journals about our journey. This collaboration has successfully supported the educational needs of over 500 nursing students and providers in the state of Texas.”
Outside the health care world, VR simulations are also being used to help parents have difficult conversations with their children about substance abuse. Seo has developed an application that walks parents through different scenarios with their teens. It provides prompts from a surly teen avatar to help them prepare and practice potential difficult conversations with their children.
“We collaborate with the Challenge of Tarrant County and the University of North Texas College of Nursing. While they provide an educational program for parents, we contribute by offering an AI-driven system that allows them to practice interactions with a virtual avatar.”
Bringing Memories to Life
A recent program Seo is working on is exploring ways to ease the feelings of loneliness in the elderly population and aid in supporting patients with Alzheimer’s and their families in finding a sense of normalcy and connection.
“Reminiscence therapy is commonly used in nursing homes and assisted living facilities. By integrating this approach with immersive technology, we can help residents recall meaningful memories, especially by using their old photos and applying AI to create personalized virtual environments that bring their past to life.”
Seo says they can take photos from an elderly person’s life — photos of people, pets, homes — and load those into a VR scenario that quite literally transports them to another place and time.
“Maybe they’ve moved from where they were living 30 years ago, and now they’re in the nursing home, often without familiar surroundings like their garden or personal spaces. So, utilizing old photos, we can recreate those meaningful environments in virtual reality. So for them to be like, oh, I’m there. I’m in my house, you know, like 30 years ago. I remember the fireplace was there, and my bookshelf was there. I remember I had a garden with a lot of flowers. These VR experiences not only immerse them in familiar settings but also stimulate conversation and help trigger vivid memories, enhancing emotional connection and cognitive recall.”
Whether it’s helping students find a new way to learn complicated material or helping an elderly nursing home resident remember younger days, Seo and her team are working together to make a meaningful difference in the lives of Texans and beyond with their innovative approach to integrating the virtual world into the real world.
