Course Project / CS182 Machine Learning
MOF3R: Mask-Guided 3D Product Reconstruction
Mask-Guided High-Fidelity 3D Product Reconstruction via SAM2 and 3D-Consistent Gaussian Splatting Refinement
CS182: Introduction to Machine Learning · Course Project · ShanghaiTech University · 2026
CS182
Overview
MOF3R is a segmentation-guided object-centric 3D reconstruction course project built on 3D Gaussian Splatting. It uses SAM2-generated foreground masks as semantic priors, optimizes 3DGS with a foreground-aware compound loss, and applies geometry-aware pruning to remove floating Gaussians and boundary artifacts. Experiments on CO3Dv2 show cleaner object reconstructions with sharper boundaries and fewer background artifacts than vanilla 3DGS.
Summary
- Input monocular videos are processed with COLMAP for camera parameters and SAM2 for foreground masks.
- Foreground masks guide 3DGS training so optimization focuses on target objects instead of cluttered backgrounds.
- Post-training refinement combines mask consistency, local density, anisotropy filtering, and adaptive shrinkage.
- The project evaluates object-centric reconstruction quality on representative CO3Dv2 sequences.
This project was completed as a CS182 course project by Zidong Song, Boyang Zhou, and Zian Chen.
Method
The method is organized as a compact three-stage pipeline: camera and mask preprocessing, mask-guided 3DGS optimization, and geometry-aware pruning for residual artifacts.
Method
Preprocessing with COLMAP and SAM2
- Extract frames from the input video sequence.
- Estimate camera intrinsics and extrinsics with COLMAP.
- Generate foreground masks with SAM2 using a first-frame prompt.
Method
Mask-guided 3DGS optimization
- Use SAM2 masks as foreground semantic priors.
- Apply mask-constrained L1 supervision to focus photometric training on the object.
- Use composite-image SSIM to preserve structural consistency near boundaries.
Method
Geometry-aware pruning
- Project Gaussians into visible views and vote with foreground masks.
- Analyze local KNN density and anisotropy to identify outlier primitives.
- Shrink or remove uncertain Gaussians to smooth silhouettes while preserving object detail.
Mask-Guided Compound Loss
The compound objective combines a mask-constrained L1 term with composite-image SSIM so foreground reconstruction improves without encouraging the model to fit background clutter.
- SAM2 foreground masks provide semantic supervision.
- Mask-constrained L1 focuses pixel loss on the object.
- Composite-image SSIM stabilizes boundary structure.
- The objective reduces background fitting during 3DGS optimization.
Geometry-Aware Multi-Metric Pruning
Multi-view mask consistency
- Project each Gaussian center into visible camera views.
- Check whether the projections fall inside foreground masks.
- Remove Gaussians that consistently vote as background across valid views.
Geometry-Aware Multi-Metric Pruning
Density and anisotropy filtering
- Use K-nearest-neighbor neighborhoods to estimate local density.
- Identify low-density outliers and excessive anisotropy.
- Target stretched structures and trailing artifacts near object boundaries.
Geometry-Aware Multi-Metric Pruning
Adaptive shrinkage
- Avoid deleting all uncertain boundary candidates immediately.
- Gradually reduce scale and opacity for lower-confidence primitives.
- Preserve fine geometry while smoothing noisy silhouettes.
Experiments and Results
Experiments use CO3Dv2 object sequences, compare against original 3D Gaussian Splatting, and report foreground-mask PSNR, SSIM, and LPIPS. The numbers below are course project results from representative real object sequences rather than publication claims.
Summary
- PSNR improves from 19.07 dB for original 3DGS to 23.56 dB for MOF3R.
- SSIM increases from 0.592 to 0.935, indicating stronger foreground structural consistency.
- LPIPS decreases from 0.629 to 0.120, matching the reduction in visible perceptual artifacts.
Visual Results
A single visual section collects the core pipeline, qualitative reconstruction comparison, and quantitative comparison from the project report.
