arxiv:2607.13431

Discrete Diffusion Models: A Unified Framework from Tokenization to Generation

Published on Jul 15

· Submitted by

Ye Yuan on Jul 16

AAAAA Academia Attractions

Upvote

Authors:

Abstract

Discrete denoising diffusion models (DDMs) have recently emerged as a compelling alternative to autoregressive (AR) modeling for discrete data, offering parallel generation and iterative global refinement capabilities. Unlike continuous diffusion, where the state space is fixed, DDMs are fundamentally shaped by how the discrete state space is constructed: the tokenization scheme, the vocabulary topology, and domain-specific structural alphabets. This work introduces a unified conceptual framework that views discrete diffusion models through the construction of the underlying discrete state space. Within this framework, existing formulations, including transition-matrix, masking/absorbing-state, and score/ratio-based approaches, emerge as different instantiations of a common design space. The framework further exposes common design trade-offs across training objectives, inference algorithms, scaling behavior, systems optimization, and evaluation protocols, suggesting several promising directions for future research.

View arXiv page View PDF GitHub 7 Add to collection

Community

stevenyuan666

Paper submitter about 23 hours ago

Why is applying diffusion to text, proteins, and molecules so much harder than applying it to images?

I'm excited to share our new survey paper: "Discrete Diffusion Models: A Unified Framework from Tokenization to Generation."

Discrete diffusion has emerged as a compelling alternative to autoregressive generation: it enables parallel decoding and, unlike left-to-right models; it can revise earlier decisions in light of later context. But it comes with a catch that continuous diffusion never faces: in image space, the state space is fixed and Gaussian noise has an obvious geometry. In categorical space, there is no default notion of a "small perturbation."

Our central thesis: the construction of the discrete state space, tokenization, is not a preprocessing detail. It is the primary design axis. It determines the topology of corruption, the difficulty of the denoising task, downstream controllability and validity, and the computational cost.

What we contribute:
🔹 A tokenization-centric lens that unifies three token families: semantic (subwords), quantized (VQ codebooks), and natural alphabets (amino acids, nucleotides, atom types)
🔹 A four-component framework — corruption operator, denoiser parameterization, training objective, sampler — under which D3PM, masked diffusion, SEDD, and discrete flow matching are all special cases, often differing in only one component
🔹 A cross-domain map spanning text/code, multimodal generation, proteins, genomics, molecules and graphs, planning and agents, and tabular data
🔹 Treatment of scaling, systems, and evaluation as part of the design space rather than afterthoughts

A recurring theme: AR and diffusion are unlikely to be winner-take-all. The most effective systems we see are hybrids: AR planning with diffusion refinement, or diffusion editing atop AR backbones.

Curated paper list: https://github.com/AAAAA-Academia-Attractions/Discrete-Diffusion
Our paper link: https://arxiv.org/abs/2607.13431

Huge thanks to my collaborators Barry Li, Rui Song, ZEYU LI, Haochen Liu, Xiangyu Kong, Zixuan Dong, Linfeng Du, Zipeng Sun, Weixu Zhang, Jiaxin Huang, Changjiang Han, Yonghan Yang, Zichen (Danny) Zhao, Xiuyuan Hu, Haolun Wu, Yankai Chen, Fengran Mo, Jikun Kang, Bowei He, across McGill University, Mila, MBZUAI, University of Cambridge, University of Toronto, Salesforce, Rochester Institute of Technology, Tsinghua University, and University of Illinois Chicago. And special thanks for the guidance and support from Professor Philip Yu and Professor Steve Liu.

Feedback and disagreement very welcome.