Textured Gaussians for Enhanced 3D Scene Appearance Modeling

CVPR 2025

Project Page | Paper

Brian Chao^1,2, Hung-Yu Tseng¹, Lorenzo Porzi¹, Chen Gao¹, Tuotuo Li¹, Qinbo Li¹, Ayush Saraf¹, Jia-Bin Huang^1,3, Johannes Kopf¹, Gordon Wetzstein², Changil Kim¹

¹Meta, ²Stanford University, ³University of Maryland, College Park

Disclaimer

This is an unofficial reimplementation of the paper Textured Gaussians for Enhanced 3D Scene Appearance Modeling that highlights the key components of the paper, including:

• Ray-Gaussian intersection
• Texture Gaussians UV mapping
• Generaralized Gaussian apperance model with spatially-varying colors and alpha values

Major differences between this codebase and the original implementation include:

• Exact 2DGS rasterization is used instead of flattening 3D Gaussians and using 3DGS rasterization as used in the original implementation. This is a more accurate way of mapping 2D textures to Gaussians.
• The 3DGS-MCMC densificaiton strategy is used to flexibly control the number of Gaussians in the model.

The original code that was used for the experiments in the paper was built on top of a Meta internal codebase and therefore is not publicly available. This codebase is not an official release from Meta and is not maintained by Meta.

To facilitate open-source research and ease of use, this codebase is built on top of the gsplat software library. All features from gsplat==1.4.0 are preserved in this codebase, so you can use this same codebase to train 2DGS, 3DGS, and Textured Gaussians.

Setup

The code was tested on:

• OS: Ubuntu 22.04.5 LTS
• GPU: NVIDIA RTX A6000
• Driver Version: 560.35.03
• CUDA Version: 12.6
• nvcc: 11.5
• Python Version: 3.12.7
• Torch Version: 2.4.1

To create a conda environment with the same dependencies as the codebase, please run:

conda env create -f environment.yml

If the above command does not work, please create a new conda environment and install the dependencies manually (especially torch and torchvision).

conda create -n textured_gaussians python=3.12.7
conda activate textured_gaussians

Then, clone this repository and install the Textured Gaussians codebase by running:

git clone https://github.com/bchao1/textured-gaussians.git
cd textured-gaussians
python -m pip install -e . # install in editable mode for development

Dataset

• You can download the NeRF synthetic dataset here and the Mip-NeRF 360 dataset here.
• Custom data loaders that support NeRF synthetic dataset and COLMAP dataset formats are defined in examples/datasets/. You can easily extend the code to support your own dataset.

Optimization

Textured Gaussians are optimized in two stages: the 2DGS pre-training stage and the Textured Gaussians refinement stage.

Please refer to the scripts/ folder for examples on how to run the code. The default trainer Python script is examples/simple_trainer_textured_gaussians.py that supports both 2DGS and Textured Gaussians optimization. The script also automatically computes the image quality metrics (PSNR, SSIM, LPIPS) and saves them to a json file.

Stage 1: 2DGS Pre-training

cd ../examples
python simple_trainer_textured_gaussians.py mcmc \
    --data_dir DATASET_DIR \
    --result_dir RESULT_DIR \
    --dataset "blender" \ # "colmap" for Mip-NeRF 360
    --init_extent 1 \ # remove for Mip-NeRF 360
    --init_type "random" \ # "sfm" for Mip-NeRF 360
    --background_mode "white" \ # remove for Mip-NeRF 360
    --model_type=2dgs \
    --init_num_pts MAX_GAUSSIANS \
    --strategy.cap-max MAX_GAUSSIANS \
    --alpha_loss \ # remove for Mip-NeRF 360
    --texture_resolution 50 \
    --port 6070

Stage 2: Textured Gaussians Refinement

After 2DGS pre-training, texture maps are added to the model and the model is refined with the Textured Gaussians optimization.

cd ../examples
python simple_trainer_textured_gaussians.py mcmc \
    --data_dir DATASET_DIR \
    --pretrained_path PRETRAINED_PATH \ # path to the 2DGS pre-trained model
    --result_dir RESULT_DIR \
    --dataset "blender" \ # "colmap" for Mip-NeRF 360
    --init_extent 1 \ # remove for Mip-NeRF 360
    --init_type "pretrained" \
    --background_mode "white" \ # remove for Mip-NeRF 360
    --model_type=textured_gaussians \
    --init_num_pts MAX_GAUSSIANS \
    --strategy.cap-max MAX_GAUSSIANS \
    --strategy.refine-start-iter=1000000000000 \
    --alpha_loss \ # remove for Mip-NeRF 360                
    --textured_rgb \ # remove if no RGB texture is used
    --textured_alpha \ # remove if no alpha texture is used
    --texture_resolution 50 \
    --port 6070

Results

Examples of 2DGS and Textured Gaussians optimization results are shown below. The number of Gaussians used in both models is 10000 and the RGBA textured map resolution for Textured Gaussians is 50x50.

Ground Truth	2DGS	Textured Gaussians

License

This codebase is Apache 2.0 licensed. Please refer to the LICENSE file for more details.

Citation

If you find our work useful, please cite:

@inproceedings{chao2025texturedgaussians,
    title={Textured Gaussians for Enhanced 3D Scene Appearance Modeling},
    author={Brian Chao and Hung-Yu Tseng and Lorenzo Porzi and Chen Gao and Tuotuo Li and Qinbo Li and Ayush Saraf and Jia-Bin Huang and Johannes Kopf and Gordon Wetzstein and Changil Kim},
    year={2025},
    booktitle={CVPR}
}