Fuzzy Metaballs: Approximate Differentiable Rendering with Algebraic Surfaces
Leonid Keselman and Martial Hebert
ECCV 2022 (Oral)
Shape From Silhouette Reconstruction from a short cell phone video, showing reconstructed masks.
Abstract
Differentiable renderers provide a direct mathematical link between an object's 3D representation and images of that object. In this work, we develop an approximate differentiable renderer for a compact, interpretable representation, which we call Fuzzy Metaballs. Our approximate renderer focuses on rendering shapes via depth maps and silhouettes. It sacrifices fidelity for utility, producing fast runtimes and high-quality gradient information that can be used to solve vision tasks. Compared to mesh-based differentiable renderers, our method has forward passes that are 5x faster and backwards passes that are 30x faster. The depth maps and silhouette images generated by our method are smooth and defined everywhere. In our evaluation of differentiable renderers for pose estimation, we show that our method is the only one comparable to classic techniques. In shape from silhouette, our method performs well using only gradient descent and a per-pixel loss, without any surrogate losses or regularization. These reconstructions work well even on natural video sequences with segmentation artifacts.
Demo
Shape from Silhouette: using 32 random views and gradient descent optimization over a 400 parameter model.
Shape from Silhouette: CO3D sequences with noisy input masks. Optimization is about a minute on laptop CPU.
Pose Estimation Results
Reported as the geometric mean of rotation and translation error. Results are mean ± interquartile range.| Parameters | Noise-Free Error | Noisy Error | |
|---|---|---|---|
| Initialization | 20.2 ± 18 | 20.2 ± 18 | |
| Pulsar | 1,200 | 20.2 ± 18 | 20.2 ± 18 |
| PyTorch3D Point Cloud | 1,200 | 18.5 ± 16 | 18.4 ± 16 |
| PyTorch3D SoftRas Mesh | 750 | 14.9 ± 15 | 17.0 ± 17 |
| Equal Fidelity ICP (Plane) | 1,200 | 10.8 ± 12 | 8.2 ± 3.3 |
| Equal Fidelity ICP (Point) | 1,200 | 7.6 ± 9.9 | 8.7 ± 6.6 |
| High Fidelity ICP (Plane) | 120,000 | 8.2 ± 0.8 | 8.0 ± 3.6 |
| High Fidelity ICP (Point) | 120,000 | 6.2 ± 3.7 | 6.8 ± 3.3 |
| Fuzzy Metaballs | 400 | 4.0 ± 1.5 | 4.2 ± 2.1 |
Shape From Silhouette (SFS) Results
Cross-entropy silhouette loss on 32 novel viwes for 10 sample models. Runtimes were the average per model on CPU. Results show μ ± σ.| Time (s) | Noise-Free Error | Noisy Error | |
|---|---|---|---|
| Voxel Carving | 82 | 0.31 ± 0.10 | 1.12 ± 0.37 |
| PyTorch3D Point Cloud | 185 | 0.08 ± 0.08 | 0.10 ± 0.08 |
| PyTorch3D SoftRas Mesh | 3,008 | 0.06 ± 0.05 | 0.07 ± 0.05 |
| NeRF | 7,406 | 0.03 ± 0.02 | 0.062 ± 0.06 |
| Fuzzy Metaballs | 68 | 0.04 ± 0.02 | 0.055 ± 0.02 |
Paper
