arxiv:2610.20818

Intern Lumina U2: A Multi-Codebook Diffusion Large Language Model for Omni-Visual Understanding and Image Generation

Published on Oct 14

· Submitted by

Zongyang (Zane) QIU on Oct 16

Shanghai AI Laboratory

Upvote

Authors:

Zongyang Qiu ,

Abstract

Intern Lumina U2 is a diffusion language model that uses multi-codebook discrete visual tokens to unify image, video, and 3D generation with text in a single backbone, achieving high performance and faster inference.

Generated by thinkingmachines/Inkling-Small

Unified multimodal generation covers discrete, continuous, and hybrid paradigms. Discrete approaches with single-codebook visual tokenizers align naturally with text modeling but have limited representational capacity. Continuous and hybrid approaches retain richer visual representations, but they model text and images in heterogeneous spaces with different training objectives, potentially introducing optimization conflicts between modalities with misaligned representation spaces. These choices expose a central question: how to efficiently model high-capacity visual representations alongside textual content in a shared optimization space within a single backbone. We present Intern Lumina U2, to our knowledge the first diffusion language model to represent multi-codebook visual tokens in a fully discrete space. Each visual position is represented by a group of codes from complementary codebooks, increasing its representational capacity while preserving compatibility with discrete text modeling. At the input, the embeddings are concatenated at each position; at the output, a lightweight multi-codebook autoregressive head predicts the codes within each position sequentially, while the diffusion backbone processes all positions in parallel. Built on the unified 4D representation of AToken, Intern Lumina U2 supports image, video, and 3D inputs within a single backbone and achieves leading performance among unified models in visual understanding, image generation, and image editing. We further build a full-stack training, inference, and evaluation system on both Ascend and NVIDIA platforms to verify the cross-platform effectiveness of our model. We also improve the inference efficiency through a stability-aware sampling strategy, which combines confidence-plateau-guided unmasking approach with drift-triggered cache refreshing mechanisms, accelerating image generation by 4.45x.

View arXiv page View PDF Project page GitHub 75 Add to collection

Community

Zane-QIU

Paper author Paper submitter about 16 hours ago

🌟 Intern Lumina U2: A Multi-Codebook Diffusion Large Language Model for Omni-Visual Understanding and Image Generation

Intern Lumina U2 is a single unified model for image / video / 3D understanding, text-to-image generation and image editing. It is a 16.9B MoE diffusion LLM with only 1.4B activated parameters, and to our knowledge the first dLLM that models multi-codebook visual tokens in a fully discrete space.

✨ Highlights

One model, omni-visual. Understanding, generation and editing over images, videos and 3D assets share one backbone, one token interface and one masked-diffusion objective, with no separate continuous generator.
Multi-codebook visual tokens. Each visual position carries 8 AToken codes, which adds detail without making the sequence longer. Positions are decoded in parallel by the diffusion backbone, and a lightweight AR head predicts the codes within each position.
Fast sampling. Plateau-Halton sampling with V-verify (PHV) gives a 4.45× end-to-end speedup for image generation.
NVIDIA + Ascend. Full-stack training, inference and evaluation on 1,024 Ascend 910B NPUs, with an NVIDIA run for cross-platform validation.

🏆 Results

Image understanding: best or tied-best on 6 of 10 benchmarks, and beats the same-backbone LLaDA2.0-Uni on 8 of 10. CharXiv-DQ 85.9 · HallusionBench 64.6 · MathVision 36.5 · DynaMath 63.6
Video / 3D: best on all three video benchmarks (Video-MME 56.7 · MLVU 62.1 · MVBench 61.8), and 3D MM-Vet 50.5, ahead of the 3D specialists ShapeLLM and PointLLM
Text-to-image: best among unified models on DPG-Bench (87.79), UniGenBench (86.11) and TIIF-Bench (88.57 / 87.08 short / long), and ties the best on GenEval (89). Outperforms FLUX.1-dev, and surpasses Qwen-Image on GenEval, UniGenBench and TIIF-Bench with ~14× fewer activated parameters
Scientific diagrams: best Local score on VeriphyT2IBench (76.58, above GPT-Image-1 and Gemini 2.5 Flash Image), and the highest GenExam-Physics relaxed score among general-purpose unified models (44.8)

Questions and feedback are very welcome! 🙌