dflow_2b / MANUAL.md
data-archetype's picture
dflow_2b 0.215: MOSS layout support
0957ad9 verified
|
Raw History Blame Contribute Delete
50 kB

dflow_2b Manual

Model card Β· Example gallery Β· Technical report Β· Release notes

Contents

  1. Installation
  2. A first image
  3. Prompts
  4. JSON layouts
  5. Image sizes
  6. Sampling
  7. Guidance
  8. Self-attention gain
  9. Python API reference
  10. ComfyUI node reference
  11. Troubleshooting

Installation

dflow_2b needs Python 3.12 or later, PyTorch 2.13 with CUDA, and an NVIDIA GPU with bfloat16 support. Install a CUDA build of PyTorch first; the PyTorch installation selector gives the command for your system.

The repository holds the denoiser weights, the text encoder, the dflow Python package and the ComfyUI nodes. The autoencoder, dinac3_96, downloads into the Hugging Face cache the first time the model is loaded.

ComfyUI

The repository is a ComfyUI custom-node folder. Download it into custom_nodes and install its requirements into ComfyUI's Python environment:

cd ComfyUI/custom_nodes
hf download data-archetype/dflow_2b --local-dir dflow_2b
pip install -r dflow_2b/requirements.txt

Restart ComfyUI. The nodes appear under the DFlow category. Start ComfyUI with --preview-method auto for live previews.

Python

Download the repository and install the package from it. The install is editable, so Python runs the code in that folder:

hf download data-archetype/dflow_2b --local-dir dflow_2b
pip install -e dflow_2b

diffusers is needed only for the optional diffusers wrapper.

Compilation

The denoiser, the text encoder and the autoencoder are compiled with torch.compile by default. The first generation takes roughly 30 to 45 seconds longer while the graphs are built.

To turn compilation off, pass compile=False in Python or switch compile_model off in the DFlow Loader node. The model then starts at once and each image takes longer. On an RTX 4090 a 50-step 832Γ—1216 image takes about 10 seconds compiled and about 13 seconds eagerly, with a peak of about 7 GB of VRAM.

A first image

In ComfyUI

Load dflow_2b/comfyui/workflows/dflow_default.json and queue it. It uses the default preset. You can also drop any PNG from the gallery onto the canvas: each one carries the workflow that made it.

The default graph has ten nodes, connected as follows:

  • DFlow Loader outputs the model, the text encoder and the autoencoder.
  • DFlow Text Encode takes the text encoder and the prompt.
  • DFlow Guider takes the model and the encoded prompt.
  • DFlow Empty Latent sets the image size.
  • DFlow Sampler feeds DFlow Time Schedule, which also takes the latent and outputs the sigmas and the sampler.
  • RandomNoise sets the seed.
  • SamplerCustomAdvanced takes the noise, the guider, the sampler, the sigmas and the latent.
  • VAEDecode takes the result and the loader's autoencoder, and SaveImage stores the image.

RandomNoise, SamplerCustomAdvanced, VAEDecode and SaveImage are ComfyUI's own nodes. The node reference lists every input.

In Python

from dflow.inference import GenerationSettings
from dflow.pipeline import DFlowPipeline

pipe = DFlowPipeline.from_pretrained("dflow_2b")
settings = GenerationSettings(width=832, height=1216, seed=7)
image = pipe("A lighthouse on a cliff at dusk.", settings).images[0]
image.save("lighthouse.png", pnginfo=pipe.png_info(image))

"dflow_2b" is the folder downloaded during installation. GenerationSettings() with no arguments is the default preset, and the seed fixes every random draw of the call.

Prompts

Plain captions

Natural descriptive captions work best. Write what a careful caption of the finished image would say: the subject, what it is doing, the setting, the light and the medium (a photograph, an oil painting, a movie still). Full sentences work better than lists of keywords, and a long, detailed prompt is fine. The limit is 6,144 tokens.

Quoted text for lettering

To put words in the image, write them in double quotes inside the caption:

A neon sign in a rainy street that reads "OPEN LATE".

Everything inside double quotes is encoded one character at a time, so the model sees the exact spelling. Use double quotes only for text that should be visible in the image. To place text precisely, use a JSON layout.

Prompt emphasis

A phrase can be given more or less weight with the markup (phrase:weight):

A portrait of a woman in a (red:1.4) coat, (blurred background:0.7).

The weight changes how much attention the rest of the model pays to the tokens of that phrase. A weight of 1 is neutral, a weight above 1 strengthens the phrase and a weight between 0 and 1 weakens it.

The markup follows a few rules:

  • The weight is the number after the last colon of a parenthesised group. The phrase itself may contain colons.
  • Groups can be nested. An inner weight replaces the outer one for its own words.
  • Parentheses without a numeric weight are ordinary text, and \( and \) always are.

Two settings control how a weight is applied. Let ww be the weight and ΞΊ\kappa the strength.

The mapping turns a positive weight into a bias added to the attention logits of the weighted tokens. linear adds 2ΞΊ(wβˆ’1)2\kappa(w-1). log adds ΞΊln⁑w\kappa \ln w, which at strength 1 multiplies the attention those tokens receive by exactly ww. The linear mapping is the stronger of the two: at strength 1, a weight of 2 multiplies the attention by about 7.

The strength scales every bias. A strength of 0 turns emphasis off without removing the markup.

A weight of 0 or below works differently. It scales the content the tokens contribute by 1+ΞΊ(wβˆ’1)1 + \kappa(w-1). At strength 1 a weight of 0 mutes the phrase and a weight of βˆ’1 subtracts it.

The weights act in every attention layer of the denoiser. The encoder switch applies them inside the text encoder as well.

Setting Default Python (EmphasisSettings) ComfyUI (DFlow Text Encode)
Emphasis on yes emphasis=EmphasisSettings(...), or None to read prompts verbatim emphasis
Mapping linear mapping emphasis_mapping
Strength 1.0 strength emphasis_strength
Encoder on encoder emphasis_encoder

Emphasis applies to plain-text prompts. Negative prompts and JSON layouts are encoded exactly as written: in a layout, parentheses and colons in captions or text stay literal and carry no weight.

Negative prompts

A negative prompt is read by classifier-free guidance (CFG) and by the contrastive modes of path-drop guidance, both described under Guidance. With those modes, an empty negative prompt selects the model's learned null text. The default guidance conditions its reference prediction on the prompt itself and ignores the negative prompt.

JSON layouts

A layout prompt describes an image as a canvas with a list of boxes. It gives control over where things are and over the exact text that appears. Layouts suit posters, covers, title cards and other designed images.

The stored format

A layout is a JSON object with exactly three keys: summary, frame and items.

summary describes the image as a whole.

Field Type Meaning
meta list of strings Tags for the kind of image, such as concert_poster
canvas string The canvas type; single in the examples here
caption string A caption of the whole image

frame describes the canvas: the overall arrangement, style and colours.

Field Type Meaning
caption string How the image is composed
tags list of strings Style and composition tags
palette list of strings Main colours as hex codes

items is a list of boxes, one per element.

Field Type Meaning
label string character, object or text
box_2d four integers The box as [y0, x0, y1, x1] on a 0 to 1000 scale
text string The exact string to render; required for text items
caption string A description of this element
tags list of strings Tags for this element
palette list of strings Its colours as hex codes

Boxes are measured from the top-left corner of the image. y0 and y1 are the top and bottom edges as thousandths of the image height, and x0 and x1 are the left and right edges as thousandths of the width. The vertical coordinate comes first.

Optional fields

The required fields are the three top-level keys and, for each item, label and box_2d, plus text on a text item. Every other field can be left out: training dropped the optional fields at random. Captions, tags and palettes give the model more to work with, and a layout with labels and boxes alone still works.

Unknown keys raise an error.

Compiling a layout to a size

Before encoding, a layout is compiled for one image size: the frame becomes a first item, labelled panel, that covers the whole canvas, and every box is converted from the 0 to 1000 scale to integer pixels measured from the centre of the image. For an image YY pixels high, the top edge y0y_0 and bottom edge y1y_1 of a box become

y0β€²=⌊y0 Y1000βŒ‹βˆ’βŒŠY2βŒ‹,y1β€²=⌈y1 Y1000βŒ‰βˆ’βŒŠY2βŒ‹, y_0' = \left\lfloor \frac{y_0\,Y}{1000} \right\rfloor - \left\lfloor \frac{Y}{2} \right\rfloor, \qquad y_1' = \left\lceil \frac{y_1\,Y}{1000} \right\rceil - \left\lfloor \frac{Y}{2} \right\rfloor,

and the left and right edges are computed the same way from the width. At 832Γ—1216 the panel box is [-608, -416, 608, 416]. The result is one line of JSON with two fields naming the convention (abs_centralized, yxyx), in the format of the MOSS-Image layout prompts.

The compiled boxes are in pixels, so a compiled layout belongs to one image size: the same stored layout compiles to different numbers at 832Γ—1216 and at 1216Γ—832. A compiled layout whose boxes do not fit the image size raises an error.

Keep layouts in the stored format, which is independent of size, and let the code compile them:

  • In ComfyUI, the DFlow Layout Encode node takes the stored JSON together with width and height. Set them to the size of the latent you sample. The guider stops with a message when they differ.
  • In Python, pass the stored JSON as the prompt. The pipeline compiles it at the size in GenerationSettings. The function that does this is dflow.inference.resolve_prompt, and dflow.layout.compile_layout returns the compiled text if you want to inspect it.

MOSS-Image layouts

The model reads the compiled form, which is the layout prompt format of MOSS-Image. MOSS-Image layouts work wherever a stored layout does, and are converted as follows:

  • MOSS-Image layout prompts (bbox in pixels from the centre, abs_centralized, yxyx), such as the output of ComfyUI's MOSS-Image Layout Prompt node, keep their items. Every layout the model was trained on began with a full-canvas panel item, so a prompt without one gets a bare panel (label and box only) put first. Every box must fit the image size.
  • MOSS-Image editable drafts (width, height, bbox in pixels from the top-left, abs, xyxy) are compiled the way MOSS-Image's compile_prompt.py compiles them, then given the panel. The draft's width and height must equal the image size.

In ComfyUI, give either to DFlow Layout Encode; DFlow Text Encode encodes a MOSS-Image layout prompt exactly as written, without the panel.

Worked example: a jazz festival poster

This layout describes an 832Γ—1216 poster with a sun, a saxophonist and two lines of text:

{"summary": {
   "meta": ["concert_poster", "flat_illustration", "mid_century_modern"],
   "canvas": "single",
   "caption": "A mid-century style jazz festival poster shows a saxophonist in silhouette against a large orange sun on a deep teal background. The title \"BLUE HARBOR JAZZ\" runs across the top in cream capitals and the line \"JULY 12 – 14, 1996\" sits at the bottom."},
 "frame": {
   "caption": "The saxophonist stands in the centre, cut off at the knees, with the sun behind his shoulders. The title fills the top fifth and the date line sits in a band along the bottom edge. Flat colour shapes with a light paper grain.",
   "tags": ["flat_color", "centered_composition", "paper_grain"],
   "palette": ["#0F4C5C", "#F28C28", "#F5E9D0", "#1B1B1B"]},
 "items": [
   {"label": "object",
    "caption": "A large flat orange circle, the sun, behind the musician.",
    "tags": ["sun", "circle"],
    "palette": ["#F28C28"],
    "box_2d": [230, 170, 720, 830]},
   {"label": "character",
    "caption": "A black silhouette of a man in a hat playing a tenor saxophone, facing right, leaning back slightly.",
    "tags": ["saxophonist", "silhouette", "hat"],
    "palette": ["#1B1B1B"],
    "box_2d": [260, 250, 880, 760]},
   {"label": "text",
    "caption": "The title \"BLUE HARBOR JAZZ\" in bold condensed cream capitals on one line.",
    "text": "BLUE HARBOR JAZZ",
    "tags": ["title", "condensed_sans", "uppercase"],
    "palette": ["#F5E9D0"],
    "box_2d": [60, 70, 180, 930]},
   {"label": "text",
    "caption": "The date line \"JULY 12 – 14, 1996\" in small spaced cream capitals.",
    "text": "JULY 12 – 14, 1996",
    "tags": ["date", "small_caps"],
    "palette": ["#F5E9D0"],
    "box_2d": [905, 300, 950, 700]}]}

Reading it from the top:

  • The summary caption describes the finished poster in one paragraph, as a plain prompt would, and quotes both lines of text.
  • The frame caption says how the canvas is arranged, and the palette lists the four colours of the design.
  • The sun comes first in items and the saxophonist second. Their boxes overlap, which is how the layout says that the figure stands in front of the sun.
  • The title box [60, 70, 180, 930] spans from 6% to 18% of the height and from 7% to 93% of the width: a wide band near the top. The text field holds the string to render and the caption describes the lettering.

The same layout ships as the workflow comfyui/workflows/dflow_layout_poster.json. In Python:

from pathlib import Path

from dflow.inference import GenerationSettings
from dflow.pipeline import DFlowPipeline

pipe = DFlowPipeline.from_pretrained("dflow_2b")
layout = Path("jazz_poster.json").read_text(encoding="utf-8")
settings = GenerationSettings(width=832, height=1216, seed=42)
image = pipe(layout, settings).images[0]

At 832Γ—1216 the sun's box [230, 170, 720, 830] compiles to [-329, -275, 268, 275]: pixels from the image centre, top edge first.

Tips

  • Write the exact characters in text, including capitals, punctuation and spaces. The model renders that string; the item's caption describes its style.
  • Keep the number of items small. A handful of well-described boxes works better than many small ones.
  • Boxes may overlap or contain one another.
  • Give each box the shape of its content. A single line of text wants a wide, short box.
  • Small text and long passages often come out wrong in this release, so prefer short strings in generous boxes.

Image sizes

Width and height must be multiples of 16, because the autoencoder maps each 16Γ—16 block of pixels to one latent cell.

The model works best at about one megapixel, the area it was trained on, in portrait, square or landscape. The default is 832Γ—1216.

Shape Sizes of about 1 MP
Portrait 832Γ—1216
Square 1024Γ—1024
Landscape 1216Γ—832

Larger images work, for example 1536Γ—832, 1216Γ—1088 or 1024Γ—1536. Composition and anatomy become less reliable as the area grows, and the self-attention gain should move towards 0. Shapes far from the trained aspect ratios are also less reliable.

Sampling

Sampling starts from noise and moves to the image in a number of solver steps. Time runs from t=1t = 1, pure noise, to t=0t = 0, the finished image. Three choices define a sampling run: the solver, the schedule of times it visits, and a shift applied to that schedule.

Solvers

A solver is the numerical method that takes each step. Some are deterministic, and some are stochastic: they add fresh noise at each step. For a stochastic solver the noise scale multiplies that noise; deterministic solvers ignore it.

Name Method Stochastic Noise scale
er_sde (default) Extended reverse-time SDE, three-stage yes 1.0; 0 makes it deterministic
er_sde_bt The same with Brownian-tree noise, so runs with different step counts follow one noise path yes 1.0
euler First-order ODE no ignored
euler_maruyama First-order reverse SDE yes 1.0 is the SDE's own noise level
dpmpp_2m DPM++ 2M, second-order multistep ODE no ignored
dpmpp_2m_sde DPM++ 2M SDE yes Acts as eta; use 0.5
abm2 Adams–Bashforth–Moulton predictor-corrector no ignored

abm2 evaluates the model twice per step, so it takes about twice as long per step as the others.

Schedules

The schedule sets where the solver's time points lie between noise and image, before the shift.

Name Spacing
beta_1_25 (default) Quantiles of a Beta(1.25, 1.25) distribution: slightly denser in the middle
beta_0_6 Quantiles of Beta(0.6, 0.6): denser at both ends
linear Evenly spaced
custom_beta Quantiles of a Beta distribution you specify by beta_mean in (0, 1) and beta_concentration above 0

For custom_beta the two Beta parameters are beta_mean Γ— beta_concentration and (1 βˆ’ beta_mean) Γ— beta_concentration. A mean above 0.5 puts more steps near the noisy end. A concentration of 2 at mean 0.5 is the linear schedule; higher values gather steps around the mean and lower values push them to the ends.

Steps

The default is 50 steps, which the gallery uses. With the default guidance each step calls the denoiser twice, once in full and once with its middle layers skipped.

Resolution-dependent shift

A larger image needs more noise to hide the same amount of structure. The model was trained with a timestep shift that grows with the image area, and the sampler applies the same shift by default.

The shift ss is expressed in log signal-to-noise ratio. For an image of width WW and height HH in pixels,

s=min⁑ ⁣(0,β€…β€Šbβˆ’ln⁑WH2562),b=βˆ’0.25. s = \min\!\left(0,\; b - \ln\frac{W H}{256^2}\right), \qquad b = -0.25 .

Each schedule time tt strictly between 0 and 1 is then moved to

tβ€²=σ ⁣(logit⁑(t)βˆ’s2), t' = \sigma\!\left(\operatorname{logit}(t) - \tfrac{s}{2}\right),

where Οƒ\sigma is the logistic function. A negative shift moves the time points towards the noisy end.

Size Area Shift ss
832Γ—1216 1.01 MP βˆ’2.99
1024Γ—1024 1.05 MP βˆ’3.02
1536Γ—832 1.28 MP βˆ’3.22
1024Γ—1536 1.57 MP βˆ’3.43

Two settings control the shift. base_log_snr_shift is bb, default βˆ’0.25. resolution_shift adds the size term and is on by default; with it off, the shift is bb alone at every size. The defaults match how the model was trained.

Guidance

Guidance sharpens an image and strengthens its adherence to the prompt by comparing two predictions at each step and pushing the result away from the weaker one.

Path-drop guidance

dflow_2b was trained so that it can also run with its 18 middle layers skipped, which gives a cheaper and weaker prediction from the same weights. Path-drop guidance (PDG) uses that weak prediction as the reference and moves the result away from it, towards the full prediction, by a scale. The technical report explains why this works.

With the full prediction vfullv_{\mathrm{full}}, the weak one vweakv_{\mathrm{weak}} and the scale ww, the guided prediction is

v=vweak+w (vfullβˆ’vweak). v = v_{\mathrm{weak}} + w\,(v_{\mathrm{full}} - v_{\mathrm{weak}}) .

A scale of 1 is no guidance, and the weak pass is then skipped. The default is 2.5.

Suffix caption bias

By default the weak pass reads the prompt less in its last six blocks: a bias of βˆ’4 is added to their attention on the prompt's tokens, on every step. This strengthens prompt following. Values closer to 0 weaken the effect and 0 turns it off; move towards βˆ’2 if colours or contrast look too strong. Below about βˆ’8 there is almost no further change. It is PdgGuidance(suffix_caption_bias=...) in Python and pdg_suffix_caption_bias on the DFlow Guider node. The technical report gives the details.

Classifier-free guidance

Classifier-free guidance (CFG) is the familiar alternative. Its reference is the full model conditioned on the negative prompt, or on the learned null text when the negative prompt is empty. The formula is the same, with that reference as vweakv_{\mathrm{weak}}. CFG is off by default. It can be used alone or combined with PDG through the modes below. With the plain PDG modes, CFG and PDG cannot share steps: give them disjoint step windows, or use a mode that includes CFG.

Modes

The PDG mode decides what the reference is at each step. "Negative text" means the negative prompt, or the learned null text when it is empty.

Mode Reference
full (default) Middle skipped, conditioned on the prompt (read with the suffix caption bias)
full_contrastive Middle skipped, conditioned on the negative text
three_quarter One image token of each 2Γ—2 group goes through the middle; conditioned on the prompt. Width and height must be multiples of 32
alternate_contrastive_positive full_contrastive on even steps, full on odd steps
alternate_pdg_first, alternate_cfg_first Contrastive PDG and CFG on alternate steps, starting with the one named
alternate_positive_pdg_first, alternate_positive_cfg_first The same with prompt-conditioned PDG
combined_cfg_pdg CFG and contrastive PDG on every step, their two corrections averaged
pdg_with_alternating_cfg Contrastive PDG on every step, with CFG averaged in on odd steps
cfg_to_pdg CFG before the PDG step window, prompt-conditioned PDG inside it

Steps are counted from 0. The modes that involve CFG need a CFG scale even when plain CFG is switched off.

In the two averaging modes the scales are extra strengths added on top of the full prediction, so a PDG or CFG value of 0 there means no contribution from that side.

Scale curves

The PDG scale can be constant or vary along the trajectory.

Curve Behaviour Parameters and defaults
Constant (default) One scale at every step scale 2.5
Ramp Starts at one scale at the noisy end, moves linearly to another over the first part of the schedule, then holds start 4.5, end 2.5, length 0.3
Endpoint curve Goes from a scale at the first step to a scale at the last, along a chosen shape noisy scale 2.0, clean scale 2.5
Bezier 1 plus a multiplier times a fixed fitted curve over time multiplier 3.0

The ramp's length is a fraction of the schedule: 0.3 with 50 steps means the scale reaches its end value at step 15.

An endpoint curve has one of four shapes:

Shape Interpolation between the two scales
linear Linear in the step index
power (step / steps) raised to power, default 3: stays near the noisy scale for longer
s_curve A logistic curve with its midpoint at s_curve_knee (default 0.5) and slope s_curve_sharpness (default 8)
linear_logsnr Linear in the log signal-to-noise ratio of each step

The ramp cannot be combined with the cfg_to_pdg mode, which switches on the step window.

Step windows

PDG and CFG each have a step window: a first and a last step, both inclusive, outside which that guidance is off. The default window covers every step. In ComfyUI a stop step of βˆ’1 means the last step.

Projected guidance (APG)

High guidance scales raise contrast and saturation. Projected guidance reduces this by splitting the guidance correction into a part parallel to the full prediction of the clean image and a part orthogonal to it, and keeping only a fraction of the parallel part. That fraction is parallel_weight: 1 is plain guidance and 0 removes the parallel part.

The projection has two forms. global projects each image as a whole. per_band splits the latent into five spatial frequency bands, from the mean colour (band 0) and the coarsest structure (band 1) up to the finest detail (band 4), and projects each band from first_band upwards, leaving the lower bands unprojected.

Projected guidance needs PDG or CFG to be on. It cannot be combined with guidance rescale or with the two averaging modes.

Guidance rescale

Guidance rescale is a simpler remedy for the same problem. It pulls the magnitude of the guided prediction back towards that of the full prediction. A value of 0 is off and 1 matches the magnitude fully.

Sparse middle steps

This is a speed option. On the chosen steps the full pass sends only one image token of each 2Γ—2 group through the middle layers, a configuration the model was trained on. percent is the share of steps that run this way, and the schedule is prefix for the first steps of the run or cyclic to spread them evenly. The default is 0, which is off. Width and height must be multiples of 32.

Defaults

Setting Default
PDG on, mode full, constant scale 2.5, all steps, suffix caption bias βˆ’4
CFG off
Self-attention gain βˆ’0.03 on the full pass, 0 on the guidance pass
Projected guidance off
Guidance rescale 0
Sparse middle steps 0%
Negative prompt empty

Recipes

Each recipe changes one thing from the defaults. The Python snippets build a GuidanceSettings object, which goes into GenerationSettings(guidance=...). They use these imports:

from dflow.guidance import (
    BezierPdgCurve,
    CfgGuidance,
    ConstantPdgScale,
    EndpointCurveKind,
    EndpointPdgCurve,
    GuidanceSettings,
    PdgGuidance,
    PdgMode,
    ProjectedGuidance,
    ProjectionMode,
    RampPdgScale,
    SparseMiddleSteps,
    SparseStepSchedule,
    StepWindow,
)

In ComfyUI, the mode, the constant scale, the step windows, CFG and the self-attention gain are on the DFlow Guider node. Curves, projected guidance, rescale and sparse steps are on DFlow Guider Advanced, which takes the guider as input.

A different PDG scale. Lower values are softer, higher values add contrast.

GuidanceSettings(pdg=PdgGuidance(scale=ConstantPdgScale(2.0)))

Contrastive PDG with a negative prompt. The negative prompt goes in GenerationSettings(negative_prompt=...), or into a second DFlow Text Encode node connected to the guider's negative input. A lower scale suits this mode.

GuidanceSettings(
    pdg=PdgGuidance(mode=PdgMode.FULL_CONTRASTIVE, scale=ConstantPdgScale(1.75))
)

A ramp from 4.5 to 2.5 over the first 30% of the schedule.

GuidanceSettings(pdg=PdgGuidance(scale=RampPdgScale(4.5, 2.5, 0.3)))

An endpoint curve, here an S-curve from 2.0 to 3.0.

GuidanceSettings(
    pdg=PdgGuidance(
        scale=EndpointPdgCurve(
            kind=EndpointCurveKind.S_CURVE, noisy_scale=2.0, clean_scale=3.0
        )
    )
)

The Bezier curve.

GuidanceSettings(pdg=PdgGuidance(scale=BezierPdgCurve(multiplier=3.0)))

PDG without the suffix caption bias.

GuidanceSettings(pdg=PdgGuidance(suffix_caption_bias=0.0))

PDG on part of the run only, here steps 0 to 39 of 50.

GuidanceSettings(pdg=PdgGuidance(window=StepWindow(0, 39)))

CFG alone.

GuidanceSettings(pdg=None, cfg=CfgGuidance(enabled=True, scale=4.0))

CFG and PDG on alternate steps. The CFG scale is set but plain CFG stays off; the mode runs the CFG steps itself.

GuidanceSettings(
    pdg=PdgGuidance(mode=PdgMode.ALTERNATE_POSITIVE_PDG_FIRST),
    cfg=CfgGuidance(enabled=False, scale=4.0),
)

Projected guidance, per band from the third band up with the parallel part removed, at a slightly higher scale.

GuidanceSettings(
    pdg=PdgGuidance(scale=ConstantPdgScale(3.0)),
    projection=ProjectedGuidance(
        mode=ProjectionMode.PER_BAND, parallel_weight=0.0, first_band=2
    ),
)

Guidance rescale.

GuidanceSettings(rescale=0.5)

Sparse middle on half of the steps, spread evenly.

GuidanceSettings(
    sparse_middle=SparseMiddleSteps(percent=50.0, schedule=SparseStepSchedule.CYCLIC)
)

Self-attention gain

The self-attention gain is a small correction to the full pass that reduces the excess contrast guidance produces. It is part of the default preset, and its best value depends on the image size.

What it does

In every image layer of the denoiser, the gain gg multiplies the attention queries by ege^{g}. A negative gain makes attention slightly more diffuse and a positive one makes it sharper. The main pass uses the gain you set and the guidance pass uses 0.

The guidance formula amplifies any difference between the two passes by the guidance scale, so a 3% softening of attention in the main pass becomes a visible reduction in contrast and in structural errors. The technical report gives the derivation.

Choosing a value

The default, βˆ’0.03, was set at the trained sizes of about one megapixel. For larger images, move the gain closer to 0 as the area grows.

Image area Example sizes Gain
up to about 1.05 MP 832Γ—1216, 1216Γ—832, 1024Γ—1024 βˆ’0.03 (default)
about 1.3 MP 1536Γ—832, 1216Γ—1088 βˆ’0.02
about 1.6 MP 1024Γ—1536 βˆ’0.015
about 2.5 MP and above 1280Γ—1920 0 to βˆ’0.01

These values are roughly linear in the logarithm of the area, from βˆ’0.03 at 1 MP to 0 at about 2.5 MP, so sizes in between can be interpolated.

At about one megapixel, a gain closer to 0 gives oversaturation and more structural defects. At larger sizes, too negative a gain gives washed-out images.

In Python the gain is GuidanceSettings(self_attention_gain=...). In ComfyUI it is self_attention_gain on the DFlow Guider node. The gain of the guidance pass can also be set, as guidance_self_attention_gain, and should stay at 0.

Python API reference

All classes below are frozen dataclasses unless noted. Their defaults are the default preset, so GenerationSettings() reproduces it.

DFlowPipeline

dflow.pipeline.DFlowPipeline loads the model and turns prompts into images.

DFlowPipeline.from_pretrained(
    path_or_repository="data-archetype/dflow_2b",
    *,
    device="cuda",
    compile=True,
    revision=None,
    cache_dir=None,
    vae_dir=None,
)
Argument Meaning
path_or_repository A local release folder, or a Hub repository id to download
device A CUDA device
compile Compile with torch.compile; False runs eagerly
revision Hub revision, for a repository id
cache_dir Hugging Face cache directory; None for the default
vae_dir A local copy of data-archetype/dinac3_96, for offline use; None downloads the pinned commit

Calling the pipeline generates images:

pipe(
    prompt,
    settings=None,
    *,
    num_images=1,
    output_type=OutputType.PIL,
    image_to_image=None,
    callback=None,
)
Argument Meaning
prompt One prompt, or a list of prompts that all have the same number of tokens
settings A GenerationSettings; None uses pipe.default_settings
num_images Images to make from a single prompt string
output_type What to return; see OutputType
image_to_image An ImageToImage, to start from existing latents
callback A latent callback, called after each solver step

A prompt may be a plain caption or a layout in any form that JSON layouts describes; a compiled layout's boxes must fit the requested size. The images of one call share the seed and differ because each gets its own noise.

The result is a DFlowOutput with three fields: images, a list of PIL images for PIL output; tensors, the RGB or latent tensor for the other output types; and settings, the settings used.

DFlowComponents

dflow.pipeline.DFlowComponents holds the loaded parts and is available as pipe.components. DFlowComponents.from_pretrained takes the same arguments as the pipeline's.

Field Meaning
engine The denoiser and text encoder. engine.sample(prompts, settings, image_to_image=None, callback=None) returns latents
vae The autoencoder. vae.decode(latents) returns RGB in about [βˆ’1, 1]; vae.encode(images) takes RGB in [βˆ’1, 1] with sides that are multiples of 16
compiled Whether the model was compiled

Latents are float32 tensors of shape [B, 96, height / 16, width / 16].

GenerationSettings

dflow.inference.GenerationSettings holds everything except the prompt that defines one generation.

Field Default Meaning
width 832 Image width, a multiple of 16
height 1216 Image height, a multiple of 16
seed 0 Seed of every random draw, from 0 to 2⁢³ βˆ’ 1
sampling SamplingSettings() Solver, schedule and shift
guidance GuidanceSettings() Guidance and gains
emphasis EmphasisSettings() Prompt emphasis; None reads prompts verbatim
negative_prompt "" Read only by CFG and the contrastive PDG modes; empty selects the learned null text

The starting noise is drawn on the CPU from the seed, so a seed gives the same noise on every GPU.

SamplingSettings

dflow.inference.SamplingSettings:

Field Default Meaning
solver Solver.ER_SDE A member of dflow.solvers.Solver
noise_scale 1.0 Noise multiplier of stochastic solvers
schedule TimeSchedule() Spacing and step count
base_log_snr_shift βˆ’0.25 The size-independent part of the shift
resolution_shift True Add the image-size term

dflow.schedules.TimeSchedule has kind (a ScheduleKind, default BETA_1_25), steps (default 50), and beta_mean and beta_concentration for CUSTOM_BETA (defaults 0.5 and 0.6).

from dflow.inference import GenerationSettings, SamplingSettings
from dflow.schedules import ScheduleKind, TimeSchedule
from dflow.solvers import Solver

settings = GenerationSettings(
    width=1024,
    height=1024,
    seed=3,
    sampling=SamplingSettings(
        solver=Solver.DPMPP_2M,
        schedule=TimeSchedule(kind=ScheduleKind.LINEAR, steps=40),
    ),
)

GuidanceSettings and its parts

dflow.guidance.GuidanceSettings:

Field Default Meaning
pdg PdgGuidance() Path-drop guidance; None turns it off
cfg CfgGuidance() Classifier-free guidance
self_attention_gain βˆ’0.03 Gain of the main pass, from βˆ’0.5 to 0.1
guidance_self_attention_gain 0.0 Gain of the guidance passes
projection ProjectedGuidance() Projected guidance (APG)
rescale 0.0 Guidance rescale, from 0 to 1
sparse_middle SparseMiddleSteps() Sparse middle steps

The parts, all in dflow.guidance:

Class Fields and defaults
PdgGuidance mode=PdgMode.FULL, scale=ConstantPdgScale(), window=StepWindow(), suffix_caption_bias=-4.0 (from βˆ’10 to 0; 0 is off)
CfgGuidance enabled=False, scale=None, window=StepWindow()
StepWindow start_step=0, stop_step=None (the last step); both inclusive
ConstantPdgScale scale=2.5
RampPdgScale start_scale=4.5, end_scale=2.5, length=0.3
EndpointPdgCurve kind=EndpointCurveKind.POWER, noisy_scale=2.0, clean_scale=2.5, power=3.0, s_curve_knee=0.5, s_curve_sharpness=8.0
BezierPdgCurve multiplier=3.0, plus the control points of the curve
ProjectedGuidance mode=ProjectionMode.OFF, parallel_weight=1.0, first_band=0
SparseMiddleSteps percent=0.0, schedule=SparseStepSchedule.PREFIX

CfgGuidance.scale must be set when CFG is enabled or when the PDG mode uses CFG, and must be left as None otherwise. Inconsistent settings raise a ValueError when the object is built, before any sampling.

EmphasisSettings

dflow.prompt_emphasis.EmphasisSettings has mapping (EmphasisMapping.LINEAR or LOG, default LINEAR), strength (default 1.0) and encoder (default True). See Prompt emphasis.

ImageToImage

dflow.inference.ImageToImage starts sampling from existing latents, noised to the time of a later schedule step.

Field Meaning
latents [B or 1, 96, h, w] latents of the generation size, for example from pipe.components.vae.encode
start_step The step of the schedule at which sampling starts, from 0 to steps βˆ’ 1

The schedule is built at its full step count. The latents are noised to the time of start_step with the seed's noise, and the remaining steps run. A later start step keeps more of the input image. Guidance step windows count from the start step.

OutputType

dflow.pipeline.OutputType:

Member Result
PIL images: PIL images that carry their generation metadata
TENSOR tensors: [B, 3, H, W] float32 RGB in [βˆ’1, 1]
LATENT tensors: undecoded [B, 96, H/16, W/16] latents

Previews during sampling

A latent callback has the signature callback(latents, completed, total). It is called after each solver step with the model's current estimate of the finished latents and the step counts.

dflow.preview.LatentRgbPreview is a fitted projection from those latents to a small RGB image, at one eighth of the image resolution.

from dflow.preview import LatentRgbPreview, preview_images

preview = LatentRgbPreview.bundled().to(pipe.components.engine.device)


def show(latents, completed, total):
    image = preview_images(preview.project(latents))[0]
    image.save(f"preview_{completed:03d}.png")


image = pipe("A lighthouse on a cliff at dusk.", callback=show).images[0]

PNG metadata

Each PIL image returned by the pipeline carries a JSON description of how it was made in image.info["dflow"]: the model and package versions, the prompt, the index of the image in its batch and the complete settings.

To keep it in the saved file, pass it to save:

image.save("out.png", pnginfo=pipe.png_info(image))

The text is stored in a PNG text chunk named dflow.

Diffusers wrapper

dflow.diffusers_pipeline.DFlowDiffusersPipeline offers the call signature of a diffusers pipeline. It requires diffusers to be installed and wraps a DFlowPipeline; the model, sampler and guidance are the same.

from dflow.diffusers_pipeline import DFlowDiffusersPipeline

pipe = DFlowDiffusersPipeline.from_pretrained("dflow_2b")
image = pipe(
    "A lighthouse on a cliff at dusk.",
    height=1216,
    width=832,
    num_inference_steps=50,
    guidance_scale=2.5,
    generator_seed=7,
).images[0]
Argument Default Meaning
prompt The prompt
negative_prompt "" Accepted only with settings, whose guidance must read it (CFG or contrastive PDG); set it in those settings
height, width 1216, 832 Image size
num_inference_steps 50 Solver steps
guidance_scale 2.5 The PDG scale
num_images_per_prompt 1 Images to generate
generator_seed 0 The seed
settings None A full GenerationSettings; when given it replaces every argument except the prompt and the image count

guidance_scale here is the PDG scale, so values suited to CFG in other models are too high. The wrapper is loaded with its own from_pretrained, as above. It does not support save_pretrained or loading through DiffusionPipeline.from_pretrained.

ComfyUI node reference

The nodes are in the DFlow category. Their defaults are the default preset.

DFlow Loader

Loads the denoiser, the text encoder and the autoencoder. Outputs: model, text_encoder, vae. The vae output works with ComfyUI's VAEDecode and VAEEncode nodes.

Input Default Meaning
source bundled bundled reads the weights in the node folder; local_dir another folder; hub a Hugging Face repository
local_dir empty Release folder, for the local_dir source
repository data-archetype/dflow_2b Repository id, for the hub source
revision empty Hub revision; empty for main
compile_model on Compile the model with torch.compile
cache_dir empty Hugging Face cache directory; empty for the default

DFlow Text Encode

Encodes a plain prompt. Output: conditioning.

Input Default Meaning
text_encoder From the loader
text empty The prompt
batch_size 1 Copies of the prompt; must equal the latent's batch size
emphasis on Apply (phrase:weight) markup
emphasis_mapping linear linear or log
emphasis_strength 1.0 Multiplier on every weight's effect
emphasis_encoder on Also apply the weights inside the text encoder

For a negative prompt, add a second DFlow Text Encode node and connect it to the guider's negative input. Stored layouts and MOSS-Image drafts go to DFlow Layout Encode, which takes the image size they are compiled for.

DFlow Layout Encode

Compiles a layout at the image size and encodes it. Output: conditioning.

Input Default Meaning
text_encoder From the loader
layout_json empty A stored or compiled layout, or a MOSS-Image layout prompt or draft
width 832 Image width the layout is for
height 1216 Image height the layout is for
batch_size 1 Copies of the prompt

width and height must equal the size of the latent that is sampled.

DFlow Guider

Sets path-drop guidance, its suffix caption bias, CFG and the self-attention gain. Output: guider.

Input Default Meaning
model From the loader
positive The prompt's conditioning
negative not connected Optional conditioning for CFG and the contrastive PDG modes
pdg_enabled on Path-drop guidance
pdg_mode full The mode
pdg_scale 2.5 Constant PDG scale; 1 is none
pdg_start_step 0 First PDG step
pdg_stop_step βˆ’1 Last PDG step; βˆ’1 for the last step
pdg_suffix_caption_bias βˆ’4 Suffix caption bias of the reference; 0 is off
cfg_enabled off Classifier-free guidance
cfg_scale 4.0 CFG scale, also used by the PDG modes that involve CFG
cfg_start_step 0 First CFG step
cfg_stop_step βˆ’1 Last CFG step; βˆ’1 for the last step
self_attention_gain βˆ’0.03 Gain of the main pass

DFlow Guider Advanced

Takes a guider from DFlow Guider and returns it with further settings. It is optional. Output: guider.

Input Default Meaning
guider From DFlow Guider
guidance_self_attention_gain 0.0 Gain of the guidance passes
pdg_curve constant constant, ramp, linear, linear_logsnr, power, s_curve or bezier
curve_noisy_scale 2.0 Endpoint curves: scale at the first step
curve_clean_scale 2.5 Endpoint curves: scale at the last step
curve_power 3.0 Exponent of the power curve
s_curve_knee 0.5 Midpoint of the s_curve
s_curve_sharpness 8.0 Slope of the s_curve
ramp_start_scale 4.5 Ramp: scale at the noisy end
ramp_end_scale 2.5 Ramp: scale after the ramp
ramp_length 0.3 Ramp: fraction of the schedule
bezier_multiplier 3.0 Height of the Bezier curve
projection off Projected guidance: off, global or per_band
projection_parallel_weight 1.0 Weight kept on the parallel part
projection_first_band 0 per_band: first projected band, 0 to 4
rescale 0.0 Guidance rescale; 0 is off
sparse_middle_percent 0.0 Percent of steps with a sparse middle
sparse_middle_schedule prefix prefix or cyclic

With pdg_curve at constant the scale is the pdg_scale of the DFlow Guider node. The other choices replace it and read only their own inputs.

DFlow Sampler

Chooses the solver. Output: sampler, which goes to DFlow Time Schedule.

Input Default Meaning
solver er_sde One of the solvers
noise_scale 1.0 Noise multiplier of stochastic solvers; eta for dpmpp_2m_sde

Changing the solver in the ComfyUI interface sets noise_scale to that solver's recommended value.

DFlow Time Schedule

Builds the schedule for the size of the latent. Outputs: sigmas and sampler.

Input Default Meaning
sampler From DFlow Sampler
latent The latent that will be sampled; gives the image size
schedule beta_1_25 One of the schedules
steps 50 Solver steps
base_log_snr_shift βˆ’0.25 The size-independent part of the shift
resolution_shift on Add the image-size term
beta_mean 0.5 custom_beta only
beta_concentration 0.6 custom_beta only

Connect both outputs to SamplerCustomAdvanced. The sampler output carries the schedule the sigmas were built from: its kind, its shift and the image size. Guidance computes the ramp and the linear_logsnr curve from them and checks the size against the latent. With a sampler taken straight from DFlow Sampler, sampling stops with an error message.

DFlow Empty Latent

Creates an empty 96-channel latent. Output: latent. ComfyUI's stock empty latent nodes produce a different latent format and cannot be used.

Input Default Meaning
width 832 Image width, a multiple of 16
height 1216 Image height, a multiple of 16
batch_size 1 Images per run

Stock nodes

Seeds come from ComfyUI's RandomNoise node. A seed gives the same starting noise in ComfyUI as in the Python package.

SamplerCustomAdvanced takes the noise, the guider, the sampler and sigmas from DFlow Time Schedule, and the latent. VAEDecode with the loader's vae output turns the result into an image, and SaveImage stores it with the workflow embedded.

Troubleshooting

"The dflow_2b weights are not in the node folder", or a file "is a Git LFS pointer". This happens after a git clone without the large files. Download them into the folder with the command the message prints, hf download data-archetype/dflow_2b --local-dir <folder>, or set the loader's source to hub.

The first image takes much longer than the following ones. The model is being compiled. This happens once per session.

Errors during compilation. torch.compile needs a working compiler toolchain and a matching PyTorch build. To check whether compilation is the cause, run eagerly: compile=False in Python, or compile_model off in DFlow Loader. Eager mode runs the same model at lower speed.

"Please enter a prompt." Empty prompts are off in the release's config.json, so a forgotten prompt fails instead of generating freely. To allow them, set "allow_empty_prompt": true under "inference" there.

Out of memory. An 832Γ—1216 image needs about 7 GB of VRAM at peak, and memory grows with the image area and the batch size. Generate one image at a time, reduce the size, and close other programs that hold GPU memory. In ComfyUI, other loaded models count against the same memory.

"The prompt has N tokens; the model takes at most 6144." Shorten the prompt. Text in double quotes and text fields of a layout use one token per character, so long quoted passages add up quickly.

"Prompts of one batch must have the same token count." A list of different prompts in one Python call must have equal token counts. Run prompts of different lengths as separate calls, or use num_images to repeat one prompt.

"This layout prompt was compiled for … the image is …" A compiled layout is being used at a different size. Compile the stored layout again at the image size: in ComfyUI, set width and height on DFlow Layout Encode to the latent's size.

"The schedule was built for … but the latent is …" The DFlow Time Schedule node is connected to a different latent from the one being sampled. Connect the same latent to both.

"Connect the sampler output of DFlow Time Schedule …" SamplerCustomAdvanced is receiving the sampler from DFlow Sampler directly. See DFlow Time Schedule.

"The latent batch is … but the conditioning holds … prompts." Set the same batch_size on the text encode node and on DFlow Empty Latent.

Large images look dark or burnt, or lose coherence. Adjust the self-attention gain for the image area.