CodecSplat: Ultra-Compact Latent Coding for Feed-Forward 3D Gaussian Splatting

Paper | Code | Models

CodecSplat is a latent coding framework for feed-forward 3D Gaussian splatting. Instead of compressing the final irregular 3D Gaussian primitives, it entropy-codes an intermediate 2D feature representation used for depth and Gaussian prediction. This keeps the scene representation compact while preserving fast feed-forward reconstruction.

Installation

The code is developed with Python 3.10 and PyTorch 2.9.1. Please refer to the official PyTorch installation guide if your CUDA version or platform differs from the example below.

conda create -y -n py310torch291 python=3.10
conda activate py310torch291

pip install torch==2.9.1 torchvision==0.24.1 torchaudio==2.9.1 \
  --index-url https://download.pytorch.org/whl/cu130
pip install -r requirements.txt

Model Zoo

Pretrained weights are available on Hugging Face:

Checkpoint	Dataset	Input views	Resolution	Notes
CodecSplat-Re10K-2view	RealEstate10K	2	256x256	Variable-rate, lmb=16-1024
CodecSplat-DL3DV-8view	DL3DV	8	256x448	Variable-rate, lmb=16-1024

After downloading a checkpoint, place it under pretrained/ or update the checkpointing.pretrained_model=... entry in the corresponding test script.

Datasets

CodecSplat is evaluated on 256x256 RealEstate10K and 256x448 DL3DV. The datasets used by the provided scripts are available from:

• RealEstate10K 360p torch format
• DL3DV-ALL-480P
• DL3DV-Benchmark

The RealEstate10K release above is already in the expected torch-chunk format. For DL3DV, convert the downloaded source data with:

mkdir -p datasets

# DL3DV benchmark/test split.
python src/scripts/convert_dl3dv.py \
  --split test \
  --input_dir /path/to/DL3DV-Benchmark \
  --output_dir datasets/DL3DV-Benchmark-480P-Processed \
  --img_subdir images_8

# DL3DV training split. Test scenes are excluded by index.
python src/scripts/convert_dl3dv.py \
  --split train \
  --input_dir /path/to/DL3DV-ALL-480P \
  --output_dir datasets/DL3DV-ALL-480P-Processed \
  --img_subdir images_8 \
  --test_index_path datasets/DL3DV-Benchmark-480P-Processed/test/index.json

# Use the DL3DV benchmark split as the test split for DL3DV evaluation.
ln -s ../DL3DV-Benchmark-480P-Processed/test datasets/DL3DV-ALL-480P-Processed/test

The resulting layout should look like:

datasets
├── DL3DV-ALL-480P
│   ├── 1K
│   │   ├── 0a1b7c20...
│   │   │   ├── images_8  # 270x480
│   │   │   └── transforms.json
│   │   └── ...
├── DL3DV-Benchmark
│   ├── 0a1b7c20...
│   │   ├── images_8  # 270x480
│   │   └── transforms.json
│   └── ...
├── RE10K-Torch-360p
│   ├── train
│   │   ├── 000000.torch
│   │   ├── ...
│   │   └── index.json
│   └── test
│       ├── 000000.torch
│       ├── ...
│       └── index.json
├── DL3DV-ALL-480P-Processed
│   ├── train
│   │   ├── 000000.torch
│   │   ├── ...
│   │   └── index.json
│   └── test -> ../DL3DV-Benchmark-480P-Processed/test
└── DL3DV-Benchmark-480P-Processed
    └── test
        ├── 000000.torch
        ├── ...
        └── index.json

Evaluation

The main codec evaluation entry point is:

# Run both datasets at the default rate points, lmb=16 and lmb=1024.
python scripts/test_codec_bitstream_rd.py all

# Run one dataset at selected rate points.
python scripts/test_codec_bitstream_rd.py dl3dv 16 128 1024
python scripts/test_codec_bitstream_rd.py re10k 16 128 1024

Equivalent single-dataset shell scripts are also provided:

# DL3DV 8-view 256x448 evaluation.
lmb=1024 bash scripts/test_dl3dv_view8_256x448.sh

# RealEstate10K 2-view 256x256 evaluation.
lmb=1024 bash scripts/test_re10k_view2_256x256.sh

Useful test flags:

• test.save_image=true: save rendered target views.
• test.save_gt_image=true: save ground-truth target views.
• test.save_input_images=true: save context/input views.
• test.save_depth=true: save predicted depths.
• test.save_gaussian=true: save reconstructed Gaussian primitives as PLY.
• test.save_video=true: save rendered target-view sequences as MP4 when the evaluation index contains consecutive frames.
• model.encoder.codec_lmb_range=[L,L]: evaluate a fixed rate point L.
• model.encoder.codec_eval_use_bitstream=true: run actual compress/decompress bitstream evaluation.

Video Rendering

CodecSplat supports video rendering through test.save_video=true.

CUDA_VISIBLE_DEVICES=0 python -m src.main +experiment=dl3dv \
mode=test \
dataset/view_sampler=evaluation \
dataset.view_sampler.num_context_views=12 \
dataset.view_sampler.index_path=assets/dl3dv_start_0_distance_100_ctx_12v_video.json \
dataset.roots=[datasets/DL3DV-Benchmark-480P-Processed] \
dataset.image_shape=[256,448] \
dataset.test_len=1 \
test.save_video=true \
test.compute_scores=false \
test.render_chunk_size=10 \
test.stabilize_camera=true \
model.encoder.codec_lmb_range=[1024.0,1024.0] \
model.encoder.codec_eval_use_bitstream=true \
checkpointing.pretrained_model=/path/to/dl3dv_codec_checkpoint.ckpt \
checkpointing.no_strict_load=true \
output_dir=outputs/codecsplat_video_smoke

Training

Training is performed in two stages.

Stage 1 trains the feed-forward Gaussian reconstruction model without the learned feature codec. Stage 2 enables the feature codec and trains the compression path.

# DL3DV 8-view, 256x448.
bash scripts/train_dl3dv_view8_256x448_stage1.sh
bash scripts/train_dl3dv_view8_256x448_stage2.sh

# RealEstate10K 2-view, 256x256.
bash scripts/train_re10k_view2_256x256_stage1.sh
bash scripts/train_re10k_view2_256x256_stage2.sh

The comments in the training scripts use notation such as 1 x 8 GPUs or 4 x 8 GPUs. This means per-GPU batch size times the number of GPUs. In our experiments, the 8-GPU setting refers to 8 NVIDIA RTX 5880 Ada 48GB GPUs. On machines with fewer GPUs, gradient accumulation can be used to approximate the same effective batch size. For example, replacing 8 GPUs with 4 GPUs can often be approximated by doubling trainer.accumulate_grad_batches.

Most provided training commands use trainer.precision=bf16-mixed. Geometry, depth probability, entropy-probability, and rasterization-sensitive operations are kept in float32 internally where needed.

The provided stage-1 training scripts warm-start from DepthSplat Gaussian splatting checkpoints. Download the corresponding checkpoints before training:

mkdir -p pretrained

# RealEstate10K 2-view 256x256 warm start.
wget https://huggingface.co/haofeixu/depthsplat/resolve/main/depthsplat-gs-base-re10k-256x256-view2-ca7b6795.pth -P pretrained

# DL3DV 2-6 view 256x448 warm start.
wget https://huggingface.co/haofeixu/depthsplat/resolve/main/depthsplat-gs-base-dl3dv-256x448-randview2-6-02c7b19d.pth -P pretrained

Stage-2 training should be initialized from the corresponding stage-1 checkpoint. Update the checkpointing.pretrained_model paths in scripts/train_*_stage2.sh to your stage-1 output.

Camera Conventions

The camera intrinsic matrices are normalized, with the first row divided by the image width and the second row divided by the image height.

The camera extrinsic matrices follow the OpenCV camera-to-world convention: +X right, +Y down, and +Z pointing into the scene.

Citation

If you find this project useful, please cite CodecSplat:

@article{yu2026codecsplat,
  title   = {CodecSplat: Ultra-Compact Latent Coding for Feed-Forward 3D Gaussian Splatting},
  author  = {Yu, Pengpeng and Jiang, Runqing and Zhang, Qi and Li, Dingquan and Wang, Jing and Guo, Yulan},
  journal = {arXiv preprint arXiv:2605.25563},
  year    = {2026}
}

Acknowledgements

This project builds on several excellent open-source projects, including DepthSplat, ReSplat, gsplat, Depth Anything V2, and lossy-vae.

We also thank the dataset authors of RealEstate10K and DL3DV. We are grateful to the authors and maintainers for making their work available.