Mixed reality surgical visualization device provides 3D imaging and data

June 23, 2021

OU Health Sciences Center announced that an advanced surgical visualization device created through a collaboration between the center and Baker Hughes energy technology company now is being used in craniofacial reconstruction surgeries.

A conversation that began at the launch party for the Oklahoma City Innovation District led to a novel collaboration between the OU Health Sciences Center and Baker Hughes. Representatives from Baker Hughes and the OU Health Sciences Center started talking that day in 2018 and soon began collaborating on a prototype device.

The work leveraged augmented reality/mixed reality technology that Baker Hughes developed to create 3D reconstructions of rock specimens from computed tomography (CT) scans. Such visualization helps geologists and oil and gas companies as they determine where to drill oil wells. That capability caught the attention of OU Health plastic and reconstructive surgeon Christian El Amm, M.D., who envisioned a headset/visor that he could wear during surgery that would allow him to both see his patient as well as 3D data that was superimposed on the patient, such as CT scans and reconstruction steps he prepared for the procedure.

Baker Hughes had advanced the technology to achieve mixed reality, which allows the person wearing the headset to see what is around him or her, as well as 3D images that appear as if they were part of real life. The technology became a virtual field visit for the oil and gas industry, providing extensive information about rock, like how porous it was.

During the year-long collaboration, the team created solutions for those challenges which, until that time, were unsolved. Using artificial intelligence techniques, they “taught” the device how to recognize features of a human face, which was necessary to be able to superimpose 3D models on the patient in real time. The team also enhanced the accuracy of that process in order to meet the precision needed for surgery, and they enabled the device to track the surgeon’s instruments in real time.

Since the official collaboration ended, El Amm has continued to develop the device, with significant input from Mohammad Abdul Mukit, Ph.D., a graduate fellow in electrical and computer engineering on the OU-Tulsa campus. Notably, they have programmed the device to respond to voice commands. That allows the surgeon to keep his focus on the patient rather than turning away to look at a computer screen or clicking a mouse. They also developed “markerless tracking,” which allows the CT scan or other images to be superimposed using artificial intelligence instead of cumbersome markers to guide the way.

El Amm compares the headset to what a fighter pilot wears – both need ready access to a lot of complex information, especially if they must make a split-second decision. El Amm has begun using the device during surgical cases to enhance the safety and efficiency of complex reconstructions. Many of his patients come to him for craniofacial reconstruction after a traumatic injury; others have congenital deformities. Thus far, he has used the device for several cases, including surgery on a patient who was born without a right ear. The system took a mirror image of the patient’s left ear, then the device overlaid it on the right side, allowing El Amm to precisely attach a reconstructed ear.

In the past, he would cut a template of the ear and aim for precision using the naked eye. In another surgical case, which required an 18-step reconstruction of the face, the device overlaid the patient’s CT scan on top of his real bones. “Each one of those bones needed to be cut and moved in a precise direction,” he said. “The device allowed us to see the bones individually, then it displayed each of the cuts and each of the movements, which allowed the surgeon to verify that he had gone through all those steps. It’s basically walking through the steps of surgery in virtual reality.”

The OU Office of Technology Commercialization is pursuing intellectual property protection for the invention. In addition, El Amm has begun partnering with his colleagues in neurosurgery, orthopedic surgery and OB-GYN to develop further applications.  

Inside OU HSC has the release