NimCritic

NimCritic estimates per-frame task progress and completion from a robot video and a task instruction.

Project page · Paper · Benchmark · Demo

Contents

Path	Contents
infer_nimcritic.py	Frame loading, the sliding window, and the command-line arguments.
nimcritic_model.py	The model: video DiT, value DiT, VAE, and text encoder.
nimcritic_step_128000.pt	Step 128000. Video expert 5.00B, value expert 1.02B, bfloat16.
base/	Wan2.2 VAE, UMT5-XXL encoder, and tokenizer. The Wan2.2 DiT is not included: the full video-expert weights are already in the .pt.

base/ contains:

• DiffSynth-Studio/Wan-Series-Converted-Safetensors/Wan2.2_VAE.safetensors
• DiffSynth-Studio/Wan-Series-Converted-Safetensors/models_t5_umt5-xxl-enc-bf16.safetensors
• Wan-AI/Wan2.1-T2V-1.3B/google/umt5-xxl/ (tokenizer)

License: Apache-2.0.

Environment

Python 3.12+. Install the inference dependencies in this directory:

uv sync

Dependencies are listed in pyproject.toml: PyTorch, transformers (4.57, below 4.58), safetensors, einops, opencv-python-headless, Pillow, numpy, and tqdm.

The model code is nimcritic_model.py. The checkpoint and base/ are in the same directory. Inference does not read any other path and does not download extra files.

Inference

Run from this directory. The defaults are nimcritic_step_128000.pt and base/ next to the script.

uv run python infer_nimcritic.py \
  --video clip.mp4 \
  --task "pick up the cup" \
  --output result.json

A video file is sampled to 2 Hz from the container frame rate (--sample-hz, default 2), and the last frame is kept. A frame sequence that is already 2 Hz uses --frames and is not resampled:

uv run python infer_nimcritic.py \
  --frames frames_dir/ \
  --task "pick up the cup" \
  --output result.json

--base is the base/ directory, not the DiffSynth-Studio/ directory inside it.

Defaults: frames are resized to 256×448 (stretch); progress is a delta relative to the condition frame of the current window and is mapped back to 0–1 before it is written; rollout_stride=1; distance aggregation; 20 steps; seed 42. Add --rollout-stride 2 to match the short-task table in the paper.

Arguments, choices, and defaults:

uv run python infer_nimcritic.py --help

Check frame sampling and arguments without loading weights:

uv run python infer_nimcritic.py --self-check

Reserve at least 24 GB of GPU memory. CPU inference is --device cpu and is slower than the GPU.

In the result JSON, progress and success are per-frame values in 0–1. sampling records the frame indices that were used.

Citation

@article{xu2026nimcritic,
  title   = {Predictive Long-Horizon Progress Modeling with Action-Aware Quantification},
  author  = {Xu, Tianchi and Zheng, Yalin and Zhu, Yangxin and Ren, Ruonan and Fu, Zheng and Wang, Yunlong and Zhong, Yuanxin},
  journal = {arXiv preprint arXiv:2610.32776},
  year    = {2026},
  url     = {https://arxiv.org/abs/2610.32776}
}

Cite LOOM as well when reporting benchmark numbers.