AI with Surgical Impact
Some of the most compelling and important research problems originate from real-world issues and challenges. Such is the case for Qixing Huang, an associate professor in Computer Science whose research spans computer vision, computer graphics, and machine learning in the development of 3D generative models. On a trip to China, while giving a talk at a hospital in his hometown, Huang met a surgeon who brought up the topic of surgically correcting hand differences – such as the presence of more than five fingers on a hand – and a research problem was born. Today, the estimated rate of congenital hand differences in newborns is between 0.4% - 0.8%. For these children, effective surgeries after birth are critical.
3D modeling plays a crucial role in the surgical management of a six-fingered hand as these models enable surgeons to meticulously plan and practice the complex procedures needed to remove the extra digit and reconstruct the remaining anatomy for optimal function and appearance. The challenge is that it is difficult to use 3D scanners to reconstruct the hands of newborns because they are moving. Video-based reconstructions guided by hand templates are viable options, but current template models depend on a fixed (read five fingered) skeletal structure.
Recently, Huang, George Pavlakos, assistant professor in the Department of Computer Science and the project's principal investigator, and Volkan Isler, also a professor in Computer Science and a core member of Texas Robotics, received funding from the U.S. National Science Foundation for their research. That research, Universal Deformable Shape Models with Varying Skeletal Structures, aims to develop a new and versatile framework for 3D generative models that can accurately capture the 3D structure of deformable objects, such as animal bodies and human hands, from standard images and videos.
These models can adapt to unique and atypical anatomies, making it possible to accurately model subjects with internal skeletal variations, rare conditions, or pathologies.
Additionally, Huang received NSF funding for a project titled Small: Joint Learning of Fracture Simulation and Fractured Object Reassembly that addresses the inverse problem of fractured object reassembly. This is the process of reconstructing a complete object from its broken fragments. While much research has focused on simulating how objects break (the forward problem), automatically putting them back together remains a difficult, largely manual, and labor-intensive task.
Together with Co-PI Etienne Vouga, associate professor in the Department of Computer Science, Huang will be developing deep learning models for reassembly, which require data for training and learning fracture simulation to generate data for reassembly. The central innovation is the joint learning of fracture simulation and object reassembly. This overcomes the major hurdle of not having enough real-world fracture data. Instead of aiming for perfect prediction of how one object breaks, the physical simulation will be trained to generate vast amounts of useful synthetic data for the reassembly algorithm.
“These awards wouldn't be possible without collaboration and complementary strength among our faculty," says Huang. “Both awards share a common theme of addressing real problems with high societal impact. From bone fracture reduction using surgical robots, to reconstructing the hands of newborns born with a congenital condition.”