Intraoperative 2D/3D Image Registration via Differentiable X-ray Rendering

Abstract: We investigate the camera pose estimation problem in the context of 2D/3D medical image registration. The application is to align 2D intraoperative images (e.g., X-ray) to a patient's 3D preoperative volume (e.g., CT), helping provide 3D image guidance during minimally invasive surgeries. We present a patient-specific self-supervised approach that uses differentiable rendering to achieve the sub-millimeter accuracy required in this context. Some of aspects of our work that may be of interest to the broader ML community include
- How do you exactly compute the rendering equation for differentiable ray tracing through a voxel grid?
- What is the optimal representation of rotations and translations when using gradient descent to optimize poses?
- What is the optimal image loss function that achieves robust image registration while still being fast enough to use in real time?

Speaker bio: Vivek is a 3rd year PhD student in Polina Golland's group broadly interested in 3D computer vision problems across science and medicine.