TeethDreamer: 3D Teeth Reconstruction from Five Intra-oral Photographs

Chenfan Xu1*
Zhentao Liu1*
Yuan Liu2
Yulong Dou1
Jiamin Wu3
Jiepeng Wang1,2
Minjiao Wang4
Dingggang Shen1,5,6
Zhiming Cui1+
1 School of Biomedical Engineering & State Key Laboratory of Advanced Medical Materials and Devices, ShanghaiTech University, Shanghai, China
2Department of Computer Science, The University of Hong Kong, Hong Kong, China
3Applied Oral Sciences & Community Dental Care, Faculty of Dentistry, The University of Hong Kong, Hong Kong, China
4Shanghai Ninth People’s Hospital, School of Medicine, Shanghai JiaoTong University, Shanghai, China
5Shanghai United Imaging Intelligence Co. Ltd., Shanghai, China
6Shanghai Clinical Research and Trial Center, Shanghai, China


Paper Code

Abstract

This flowchart illustrates our algorithm for reconstructing 3D dental models (c) from multiple intra-oral photographs (a). Initially, we synthesize multi-view images and normal maps (b) using the 3D dental model from our dataset. Subsequently, we train a diffusion network to generate these images and maps directly from the intra-oral photographs, culminating in the reconstruction of the target 3D dental models.

Orthodontic treatment usually requires regular face-to-face examinations to monitor dental conditions of the patients. When in person diagnosis is not feasible, an alternative is to utilize five intra-oral photographs for remote dental monitoring. However, it lacks of 3D in formation, and how to reconstruct 3D dental models from such sparse view photographs is a challenging problem. In this study, we propose a 3D teeth reconstruction framework, named TeethDreamer, aiming to restore the shape and position of the upper and lower teeth. Given f ive intra-oral photographs, our approach first leverages a large diffu sion model’s prior knowledge to generate novel multi-view images with known poses to address sparse inputs and then reconstructs high-quality 3D teeth models by neural surface reconstruction. To ensure the 3D con sistency across generated views, we integrate a 3D-aware feature atten tion mechanism in the reverse diffusion process. Moreover, a geometry aware normal loss is incorporated into the teeth reconstruction process to enhance geometry accuracy. Extensive experiments demonstrate the superiority of our method over current state-of-the-arts, giving the po tential to monitor orthodontic treatment remotely. Our code is availableat at https://github.com/ShanghaiTech-IMPACT/TeethDreamer.

Methodology

Results

Qualitative comparisons of reconstructed 3D teeth with other baselines, demon strating our results with complete shapes and geometric details. (GT: ground truth)

Qualitative comparisons of generated images with other baselines, where our generations are closely aligned with ground truth. (GT: ground truth)

The quantitative comparison with other baselines and ablated solutions in color image generation and teeth reconstruction.

Citation

@article{xu2024teethdreamer,
                    title={TeethDreamer: 3D Teeth Reconstruction from Five Intra-oral Photographs},
                    author={Xu, Chenfan and Liu, Zhentao and Liu, Yuan and Dou, Yulong and Wu, 
                            Jiamin and Wang, Jiepeng and Wang, Minjiao and Shen, Dinggang and Cui, Zhiming},
                    journal={arXiv preprint arXiv:2407.11419},
                    year={2024}
                  }