Generate Your Talking Avatar from Video Reference
Accepted to SIGGRAPH Asia 2026
> [!IMPORTANT] > **This is not a drop-in Wan2.1 checkpoint.** It is a TAVR transformer with 18 input channels > and its own reference and audio conditioning, and it needs four companion models at inference > time (Wan VAE, umT5, wav2vec 2.0, and a person detector plus a pose model) that are not bundled > here. A stock `diffusers` Wan2.1 pipeline will not produce correct results. > > Run it through the released inference code: **https://github.com/heygen-com/TAVR** ## Model Description TAVR replaces single-image avatar references with short video clips, enabling cross-scene generation with significantly better identity preservation. A three-stage training strategy bridges the domain gap between reference and target scenes. ## Model Details | | | |---|---| | Base model | [Wan-AI/Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B) | | Input | reference video, target still, caption, driving audio | | Output | 896x480, 25 fps, driving audio muxed in | | Sampler | UniPC, 24 steps, flow shift 5.0; text guidance 5.0, audio guidance 1.8 | | Reference frames | Flexible (default 20) | | Precision | `bfloat16` | | Hardware | one Hopper-class CUDA GPU with at least 80 GB of memory | ## Usage ```bash git clone https://github.com/heygen-com/TAVR cd TAVR python3.10 -m venv .venv && source .venv/bin/activate pip install -r requirements.txt --extra-index-url https://download.pytorch.org/whl/cu126 hf download HeyGenAI/TAVR --local-dir ./ckpt PYTHONPATH=. python infer.py \ --sample-dir samples/example1 \ --dit-ckpt ckpt/tavr_transformer.safetensors \ --ckpt-dir . \ --output-dir outputs ``` `flash_attn_3` has no PyPI wheel and is the only attention backend; it and the companion models are set up as described in the [repository README](https://github.com/heygen-com/TAVR). ## Evaluation Qualitative results and comparisons against baselines are on the [project page](https://gseancdat.github.io/projects/TAVR.html). On the cross-scene benchmark, TAVR yields the best identity similarity and an overall quality score of 16.42 against 14.13 for the next best method. ### Cross-scene benchmark data `benchmark_data.json` lists the 158 reference/target pairs of the cross-scene benchmark. The clips come from the public [TalkVid](https://github.com/FreedomIntelligence/TalkVid) dataset. Each sample has: | field | content | |---|---| | `reference`, `target` | `video_id`, `video_url`, `start_time` / `end_time` (seconds), `start_frame` / `end_frame` at the source `fps`, `width`, `height` | | `target_caption` | scene caption used as the text prompt | ## Release Progress - [x] Model weights - [x] Inference code - [x] Cross-scene benchmark data ## Citation ```bibtex @inproceedings{guo2026generate, title={Generate Your Talking Avatar from Video Reference}, author={Guo, Zujin and Ye, Zhenhui and Ren, Yi and Li, Yuanming and Chen, Ce and Hong, Zhibin and Loy, Chen Change}, booktitle={SIGGRAPH Asia 2026 Conference Papers}, year={2026} } ``` ## License Apache License 2.0, inherited from [Wan2.1-T2V-14B](https://huggingface.co/Wan-AI/Wan2.1-T2V-14B). ### Third-party weights at inference time Running TAVR additionally requires DWPose pose weights (Apache-2.0) and Ultralytics YOLO weights (AGPL-3.0). Neither is redistributed here, and neither is contained in these weights; obtain them from their own channels and observe their terms. All videos and results shown here are for research demonstration purposes only.