Skip to content
Massachusetts Institute of Technology
  • on: Dec. 20, 2023
  • in: arXiv

pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction

We introduce pixelSplat, a feed-forward model that learns to reconstruct 3D radiance fields parameterized by 3D Gaussian primitives from pairs of images. Our model features real-time and memory-efficient rendering for scalable training as well as fast 3D reconstruction at inference time. To overcome local minima inherent to sparse and locally supported representations, we predict a dense probability distribution over 3D and sample Gaussian means from that probability distribution. We make this sampling operation differentiable via a reparameterization trick, allowing us to back-propagate gradients through the Gaussian splatting representation. We benchmark our method on wide-baseline novel view synthesis on the real-world RealEstate10k and ACID datasets, where we outperform state-of-the-art light field transformers and accelerate rendering by 2.5 orders of magnitude while reconstructing an interpretable and editable 3D radiance field.

Citation

@inproceedings{charatan2023pixelsplat,
    title = { pixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction },
    author = { Charatan, David and 
               Li, Sizhe and 
               Tagliasacchi, Andrea and 
               Sitzmann, Vincent },
    year = { 2023 },
    booktitle = { arXiv },
}
  • Copy to Clipboard
  • Download