VTInstructor-8B

VTInstructor-8B is the final SoTA checkpoint of VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments (ACM MM 2026).

This release is the VP-GRPO model: Qwen3-VL-8B-Instruct + VP-Adapter, after supervised fine-tuning and then VP-GRPO refinement, which updates the VP-Adapter gates together with the language decoder while the vision tower and the VP encoder stay frozen. It generates natural-language navigation instructions from egocentric RGB trajectories in continuous environments (R2R-CE / RxR-CE).

• Paper: arXiv:2608.15284
• Code: github.com/TidalHarley/VTInstructor
• Backbone: Qwen/Qwen3-VL-8B-Instruct

What this model does

Given a first-person trajectory (RGB keyframes + discrete actions), VTInstructor writes a concise indoor navigation instruction that matches the full path. Visual Trajectory Prompts (path ribbon, turn arrows, goal marker) and a 3-channel semantic mask are injected through a lightweight VP-Adapter; the same checkpoint also works on plain RGB renders without VTP, because the trajectory context is already absorbed into the weights.

Typical uses:

1. Instruction generation on R2R-CE / RxR-CE trajectories
2. Data augmentation for downstream VLN followers (see §3 below)
3. Drop-in instruction replacement on VLN-CE .json.gz splits via the open-source generation script

This model is intended for non-commercial research on vision-and-language navigation. See License.

Results

All numbers below are copied from the ACM MM 2026 paper. Figures are the original tables.

1. Navigation instruction generation (NLG)

VTInstructor is the best model on both R2R-CE Val Unseen and RxR-CE Val Unseen, including against GPT-5.4, Gemini-3.1-Pro-Preview, and much larger open-source VLMs. Relative to the strongest baseline it gains +0.357 CIDEr on R2R-CE and +0.109 CIDEr on RxR-CE.

R2R-CE Val Unseen

Method	BLEU-1	BLEU-4	METEOR	ROUGE-L	CIDEr	SPICE
Qwen3.5-Plus	0.557	0.131	0.223	0.387	0.137	0.180
GPT-5.4	0.482	0.075	0.197	0.348	0.078	0.144
Gemini-3.1-Pro-Preview	0.631	0.166	0.209	0.392	0.203	0.167
xAI Grok 4	0.486	0.084	0.207	0.354	0.073	0.160
Claude Opus 4.6	0.484	0.084	0.190	0.337	0.068	0.119
Qwen3.5-397B-A17B	0.525	0.113	0.216	0.382	0.120	0.174
Kimi-K2.5	0.478	0.088	0.195	0.357	0.098	0.146
LLaVA-Video-7B-Qwen2	0.506	0.100	0.146	0.317	0.108	0.109
Qwen3-VL-8B-Instruct	0.524	0.085	0.175	0.334	0.147	0.137
Qwen3.5-9B	0.579	0.121	0.194	0.373	0.181	0.151
GLM-4.1V-9B-Thinking	0.490	0.080	0.156	0.321	0.130	0.115
GLM-4.6V	0.556	0.120	0.171	0.348	0.168	0.122
Qwen3-VL-8B (SFT-only)	0.720	0.282	0.232	0.485	0.484	0.201
VTInstructor (this repo)	0.765	0.320	0.263	0.511	0.560	0.245

RxR-CE Val Unseen

Method	BLEU-1	BLEU-4	METEOR	ROUGE-L	CIDEr	SPICE
Qwen3.5-Plus	0.527	0.080	0.157	0.254	0.033	0.156
GPT-5.4	0.493	0.056	0.146	0.241	0.029	0.143
Gemini-3.1-Pro-Preview	0.393	0.073	0.128	0.242	0.029	0.135
xAI Grok 4	0.430	0.049	0.181	0.249	0.014	0.144
Claude Opus 4.6	0.367	0.043	0.176	0.223	0.005	0.123
Qwen3.5-397B-A17B	0.422	0.059	0.128	0.249	0.027	0.142
Kimi-K2.5	0.432	0.071	0.195	0.259	0.011	0.145
LLaVA-Video-7B-Qwen2	0.064	0.012	0.063	0.169	0.002	0.074
Qwen3-VL-8B-Instruct	0.366	0.053	0.119	0.230	0.022	0.113
Qwen3.5-9B	0.515	0.077	0.171	0.259	0.027	0.164
GLM-4.1V-9B-Thinking	0.074	0.014	0.054	0.144	0.003	0.064
GLM-4.6V	0.489	0.065	0.143	0.242	0.027	0.145
Qwen3-VL-8B (SFT-only)	0.630	0.201	0.210	0.357	0.060	0.162
VTInstructor (this repo)	0.774	0.308	0.265	0.431	0.142	0.206

Comparison with prior speaker methods trained on discretized R2R / RxR viewpoint graphs (those numbers are not on the same continuous-env protocol):

2. Frozen VLN follower

Instructions from VTInstructor are executed by a frozen CorrectNav follower on R2R-CE Val Unseen. VTInstructor improves frozen-follower success by 14.7 percentage points over the strongest baseline, and matches or exceeds official human instructions on SR / OSR / NE.

Instruction source	SR↑	OSR↑	SPL↑	NE↓
Official human instruction	61.6	67.2	53.3	4.53
Qwen3.5-Plus	48.6	61.8	39.0	5.70
GPT-5.4	35.9	51.4	28.6	6.14
Gemini-3.1-Pro-Preview	48.5	56.4	39.9	5.59
xAI Grok 4	45.5	64.8	33.4	6.18
Claude Opus 4.6	21.7	48.8	15.2	9.56
Qwen3.5-397B-A17B	45.2	63.9	33.3	6.15
Kimi-K2.5	43.9	58.6	33.6	6.49
LLaVA-Video-7B-Qwen2	30.2	44.8	24.3	7.91
Qwen3-VL-8B-Instruct	33.4	46.8	25.9	7.09
Qwen3.5-9B	36.2	45.2	29.1	6.82
GLM-4.1V-9B-Thinking	31.1	42.9	24.4	7.37
GLM-4.6V	29.9	40.6	24.0	7.11
VTInstructor (this repo)	63.3	70.0	52.7	4.47

3. Data augmentation for a VLN follower

Training CorrectNav (LLaVA-Video-7B backbone) with human R2R-CE+RxR-CE data plus VTInstructor-generated instructions yields about +3 SR on both Val Unseen splits.

R2R-CE Val Unseen

Set	Training data	SR↑	OSR↑	SPL↑	NE↓
A	R2R-CE + RxR-CE (human)	45.1	52.3	44.6	6.20
B	Setting A + VTInstructor-generated	48.4	54.2	46.8	5.85

RxR-CE Val Unseen

Set	Training data	SR↑	OSR↑	SPL↑	NE↓
A	R2R-CE + RxR-CE (human)	41.2	51.1	39.6	8.34
B	Setting A + VTInstructor-generated	44.4	53.2	41.3	7.74

Files in this repo

File	Role
model-0000{1–4}-of-00004.safetensors	Qwen3-VL-8B weights (~8.8B params, bf16). The VP tensors also appear here because the modules are registered on the model, but the code loads the two .pt files below.
vp_encoder.pt	VP-Encoder CNN (3-channel ribbon/arrow/endpoint mask → features)
vp_adapters.pt	Gated VP-Adapter at ViT layer 7
config.json, tokenizer, preprocessor	Standard Qwen3-VL processor files
figures/	Paper overview + result tables

Both vp_encoder.pt and vp_adapters.pt ship in this repo, so pointing CKPT / EVAL_CKPT at the downloaded directory is enough — no extra files are needed.

transformers==4.57.6 is the validated version. Newer transformers may break the VP-Adapter injection into Qwen3-VL internals.

Loading only with Qwen3VLForConditionalGeneration.from_pretrained gives the language/vision backbone; to reproduce paper numbers you must attach the VP modules from the code repo.

Quick start

# 1. weights
hf download TidalYang/VTInstructor-8b --local-dir ./VTInstructor-8b

# 2. code
git clone https://github.com/TidalHarley/VTInstructor.git && cd VTInstructor
pip install -r requirements.txt

Generate instructions (RGB-only, or with VTP masks if the split has them):

CKPT=../VTInstructor-8b \
DATA_DIR=/path/to/rendered_split \
DATASET_TYPE=r2rce \
OUT_JSON=outputs/augment/r2r_style.json \
  bash generate/run_generate.sh

DATASET_TYPE=rxrce switches to the longer RxR-style narration. VP_MODE=auto (default) uses vp_masks when present and falls back to RGB otherwise. MODEL_DIR is optional — the processor files in this repo are used by default. GEN_CUDA selects GPUs (default 0); shards are merged automatically.

To reproduce the paper's benchmark numbers instead (needs rendered val_unseen splits):

EVAL_CKPT=../VTInstructor-8b \
MODEL_DIR=/path/to/Qwen3-VL-8B-Instruct \
EVAL_CUDA=0 \
  bash eval/run_eval.sh

Set EVAL_CUDA to the GPU ordinals you actually have — it defaults to all eight. Scoring additionally needs pycocoevalcap and a Java runtime on PATH (for METEOR / SPICE).

Training recipe (how this checkpoint was produced)

1. SFT of Qwen3-VL-8B-Instruct + VP-Adapter on filtered R2R-CE / RxR-CE visual trajectories
2. VP-GRPO RL with a weighted NLG reward (BLEU-1/4, METEOR, ROUGE-L, CIDEr), updating the VP-Adapter gates and the language decoder while the vision tower and the VP encoder stay frozen → this checkpoint

See the GitHub README for rendering, VTP mask construction, SFT, GRPO, and evaluation.

Citation

If you use this model, please cite:

@misc{yang2026vtinstructorvisualtrajectoryprompting,
      title={VTInstructor: Visual Trajectory Prompting for Navigation Instruction Generation in Continuous Environments},
      author={Haolin Yang and Yuxing Long and Zihan Yang and Hao Dong},
      year={2026},
      eprint={2608.15284},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2608.15284},
}

License

The code release is Apache 2.0, and these weights are distributed under the same terms.

This model was trained on data derived from R2R / R2R-CE, RxR / RxR-CE and Matterport3D, which are distributed under CC BY-NC-SA 3.0 US and the Matterport3D Terms of Use. Please respect those terms — the checkpoint is intended for non-commercial research use. The Qwen3-VL-8B-Instruct backbone license also applies.