Abstract
We introduce a method for high-quality 3D reconstruction from multi-view images. Our method uses a new point-based representation, the regularized dipole sum, which generalizes the winding number to allow for interpolation of per-point attributes in point clouds with noisy or outlier points. Using regularized dipole sums, we represent implicit geometry and radiance fields as per-point attributes of a dense point cloud, which we initialize from structure from motion. We additionally derive Barnes-Hut fast summation schemes for accelerated forward and adjoint dipole sum queries. These queries facilitate the use of ray tracing to efficiently and differentiably render images with our point-based representations, and thus update their point attributes to optimize scene geometry and appearance. We evaluate our method in inverse rendering applications against state-of-the-art alternatives, based on ray tracing of neural representations or rasterization of Gaussian point-based representations. Our method significantly improves 3D reconstruction quality and robustness at equal runtimes, while also supporting more general rendering methods such as shadow rays for direct illumination.
Visualization
A visualization of all our 3D reconstruction results is available at the interactive supplemental website.
Resources
Paper: Our paper and supplement are available on arXiv, and locally.
Code: Our code is available on Github.
Data: The data to reproduce our experiments is available on Amazon S3 for Blended MVS (train and test data, point clouds) and DTU (train and test data, point clouds).
Citation
@article{Chen:Dipoles:2024,
author = {Chen, Hanyu and Miller, Bailey and Gkioulekas, Ioannis},
title = {3D reconstruction with fast dipole sums},
year = {2024},
journal = {ACM Trans. Graph.}
}
Acknowledgments
We thank Keenan Crane, Rohan Sawhney, and Nicole Feng for many helpful discussions, and the authors of Dai et al. [2024]; Wang et al. [2023]; Li et al. [2023] for help running experimental comparisons. This work was supported by NSF award 1900849, NSF Graduate Research Fellowship DGE2140739, an NVIDIA Graduate Fellowship for Miller, and a Sloan Research Fellowship for Gkioulekas.