Benjamin Warf, a renowned neurosurgeon at Boston Children’s Hospital, stands within the MIT.nano Immersion Lab. Greater than 3,000 miles away, his virtual avatar stands next to Matheus Vasconcelos in Brazil because the resident practices delicate surgery on a doll-like model of a baby’s brain.
With a pair of virtual-reality goggles, Vasconcelos is capable of watch Warf’s avatar display a brain surgery procedure before replicating the technique himself and while asking questions of Warf’s digital twin.
“It’s an almost out-of-body experience,” Warf says of watching his avatar interact with the residents. “Perhaps it’s the way it feels to have a similar twin?”
And that’s the goal: Warf’s digital twin bridged the gap, allowing him to be functionally in two places without delay. “It was my first training using this model, and it had excellent performance,” says Vasconcelos, a neurosurgery resident at Santa Casa de São Paulo School of Medical Sciences in São Paulo, Brazil. “As a resident, I now feel more confident and comfy applying the technique in an actual patient under the guidance of a professor.”
Warf’s avatar arrived via a brand new project launched by medical simulator and augmented reality (AR) company EDUCSIM. The corporate is a component of the 2023 cohort of START.nano, MIT.nano’s deep-tech accelerator that provides early-stage startups discounted access to MIT.nano’s laboratories.
In March 2023, Giselle Coelho, EDUCSIM’s scientific director and a pediatric neurosurgeon at Santa Casa de São Paulo and Sabará Children’s Hospital, began working with technical staff within the MIT.nano Immersion Lab to create Warf’s avatar. By November, the avatar was training future surgeons like Vasconcelos.
“I had this concept to create the avatar of Dr. Warf as a proof of concept, and asked, ‘What could be the place on the planet where they’re working on technologies like that?’” Coelho says. “Then I discovered MIT.nano.”
Capturing a Surgeon
As a neurosurgery resident, Coelho was so frustrated by the shortage of practical training options for complex surgeries that she built her own model of a baby brain. The physical model accommodates all of the structures of the brain and might even bleed, “simulating all of the steps of a surgery, from incision to skin closure,” she says.
She soon found that simulators and virtual reality (VR) demonstrations reduced the educational curve for her own residents. Coelho launched EDUCSIM in 2017 to expand the range and reach of the training for residents and experts seeking to learn recent techniques.
Those techniques include a procedure to treat infant hydrocephalus that was pioneered by Warf, the director of neonatal and congenital neurosurgery at Boston Children’s Hospital. Coelho had learned the technique directly from Warf and thought his avatar could be the best way for surgeons who couldn’t travel to Boston to profit from his expertise.
To create the avatar, Coelho worked with Talis Reks, the AR/VR/gaming/big data IT technologist within the Immersion Lab.
“Plenty of technology and hardware will be very expensive for startups to access as they begin their company journey,” Reks explains. “START.nano is a method of enabling them to utilize and afford the tools and technologies now we have at MIT.nano’s Immersion Lab.”
Coelho and her colleagues needed high-fidelity and high-resolution motion-capture technology, volumetric video capture, and a variety of other VR/AR technologies to capture Warf’s dexterous finger motions and facial expressions. Warf visited MIT.nano on several occasions to be digitally “captured,” including performing an operation on the physical baby model while wearing special gloves and clothing embedded with sensors.
“These technologies have mostly been used for entertainment or VFX [visual effects] or CGI [computer-generated imagery],” says Reks, “But it is a unique project, because we’re applying it now for real medical practice and real learning.”
Considered one of the most important challenges, Reks says, was helping to develop what Coelho calls “holoportation”— transmitting the 3D, volumetric video capture of Warf in real-time over the web in order that his avatar can appear in transcontinental medical training.
The Warf avatar has synchronous and asynchronous modes. The training that Vasconcelos received was within the asynchronous mode, where residents can observe the avatar’s demonstrations and ask it questions. The answers, delivered in a wide range of languages, come from AI algorithms that draw from previous research and an intensive bank of questions and answers provided by Warf.
Within the synchronous mode, Warf operates his avatar from a distance in real time, Coelho says. “He could walk across the room, he could refer to me, he could orient me. It’s amazing.”
Coelho, Warf, Reks, and other team members demonstrated a mixture of the modes in a second session in late December. This demo consisted of volumetric live video capture between the Immersion Lab and Brazil, spatialized and visual in real-time through AR headsets. It significantly expanded upon the previous demo, which had only streamed volumetric data in a single direction through a two-dimensional display.
Powerful impacts
Warf has a protracted history of coaching desperately needed pediatric neurosurgeons all over the world, most recently through his nonprofit Neurokids. Distant and simulated training has been an increasingly large part of coaching for the reason that pandemic, he says, although he doesn’t feel it’s going to ever completely replace personal hands-on instruction and collaboration.
“But when the truth is at some point we could have avatars, like this one from Giselle, in foreign places showing people find out how to do things and answering questions for them, without the fee of travel, without the time cost and so forth, I believe it may very well be really powerful,” Warf says.
The avatar project is very essential for surgeons serving distant and underserved areas just like the Amazon region of Brazil, Coelho says. “It is a strategy to give them the identical level of education that they’d get somewhere else, and the identical opportunity to be in contact with Dr. Warf.”
One baby treated for hydrocephalus at a recent Amazon clinic had traveled by boat 30 hours for the surgery, in response to Coelho.
Training surgeons with the avatar, she says, “can change reality for this baby and might change the long run.”