API.md

# Canter API and inference parameters

[Model card](README.md) · [Example gallery](GALLERY.md) ·
[Technical report](TECHNICAL_REPORT.md)

Canter exposes a high-level image pipeline and a lower-level latent inference
engine. Configuration uses frozen dataclasses and enums.

The installed `canter` package always supplies the inference implementation.
For remote model IDs, `revision` selects checkpoint artifacts only; selecting
an older checkpoint does not load or execute the Python package bundled in
that historical repository snapshot.

## Minimal use

```python
from canter import CanterPipeline

pipe = CanterPipeline.from_pretrained("data-archetype/canter")
output = pipe("A red rally car driving around a wet forest road")
image = output.image
```

### Generation arguments

| Parameter | Default | Description |
| --- | --- | --- |
| `prompts` | required | One prompt string or a sequence containing one prompt per image. |
| `negative_prompts` | `None` | Optional negative prompt string or sequence for CFG and contrastive PDG. The count must match `prompts`. `None` uses the learned unconditional token. |
| `config` | `CanterPipelineConfig()` | Inference and output settings. |
| `initial_noise` | `None` | Optional float32 latent noise tensor with the configured batch and spatial shape. |
| `progress` | `None` | Optional callback receiving completed and total solver updates. |
| `preview` | `None` | Optional callback receiving asynchronous latent-RGB PIL images plus completed and total solver updates. |

`CanterPipeline` loads the flow-matching denoiser and text tokenizer with the
bundled
[`SmolLM2-360M`](https://huggingface.co/HuggingFaceTB/SmolLM2-360M) weights. It
compiles the selected text backend, downloads DINAC-AE-D2, generates latents,
and decodes them into images.

## Loading the pipeline

```python
from canter import CanterPipeline, TextAttentionBackend, WeightDType

pipe = CanterPipeline.from_pretrained(
    "data-archetype/canter",
    dtype=WeightDType.BFLOAT16,
    text_backend=TextAttentionBackend.DENSE,
    device="cuda",
    revision=None,
    cache_dir=None,
    compile_model=True,
)
```

### `CanterPipeline.from_pretrained`

| Parameter | Default | Description |
| --- | --- | --- |
| `path_or_repo_id` | required | A Hugging Face repository ID or downloaded model repository directory. |
| `dtype` | `WeightDType.BFLOAT16` | Weight storage dtype. Compute still uses bfloat16 AMP with explicit float32 operations. |
| `text_backend` | `TextAttentionBackend.DENSE` | Bundled text-encoder, text-refinement, and cross-attention layout. |
| `device` | `"cuda"` | CUDA device string or `torch.device`. |
| `revision` | `None` | Hugging Face checkpoint branch, commit, or immutable release tag. `None` uses the installed package's pinned default checkpoint. An explicit tag such as `"v0001"` loads those weights with the currently installed code. |
| `cache_dir` | `None` | Optional Hugging Face cache directory. |
| `compile_model` | `True` | Compile the selected inference kernels. Compilation errors are reported directly. |
| `vae` | `None` | Optional compatible `CanterVae` instance. `None` downloads DINAC-AE-D2 automatically. |

### Weight dtypes

| Enum | Value | Description |
| --- | --- | --- |
| `WeightDType.BFLOAT16` | `"bfloat16"` | Default compact release with required float32 parameter islands retained. |
| `WeightDType.FLOAT32` | `"float32"` | Full-float32 weight storage release. Model compute remains bfloat16 AMP. |

### Text backends

| Enum | Value | Description |
| --- | --- | --- |
| `TextAttentionBackend.DENSE` | `"dense"` | Compatibility default using padded tensors and standard PyTorch scaled dot-product attention throughout text encoding and denoising. |
| `TextAttentionBackend.JAGGED` | `"jagged"` | Explicit optimization using packed text, CUDA FlashAttention, and jagged NestedTensor kernels throughout text encoding and denoising. |

Only the selected backend is compiled. The backend cannot be changed after
compilation.

## Pipeline configuration

```python
from canter import (
    CanterInferenceConfig,
    CanterOutputType,
    CanterPipelineConfig,
)

config = CanterPipelineConfig(
    inference=CanterInferenceConfig(),
    output_type=CanterOutputType.PIL,
)
output = pipe("A portrait lit by a large north-facing window", config=config)
```

### `CanterPipelineConfig`

| Field | Default | Description |
| --- | --- | --- |
| `inference` | `CanterInferenceConfig()` | Latent generation settings. |
| `output_type` | `CanterOutputType.PIL` | Selected decoded or latent output representation. |

### Output types

| Enum | Result |
| --- | --- |
| `CanterOutputType.PIL` | `output.images` contains one PIL image per prompt. `output.image` returns the sole image for batch size one. |
| `CanterOutputType.TENSOR` | `output.image_tensor` contains clamped float32 RGB pixels in `[-1, 1]`. |
| `CanterOutputType.LATENT` | VAE decoding is skipped. `output.latents` contains whitened float32 model latents. |

Every output also contains the descending float32 solver schedule in
`output.schedule`.

## Live latent previews

```python
def show_preview(images, completed, total):
    images[0].show()


output = pipe(
    "A lighthouse in a winter storm",
    preview=show_preview,
)
```

The bundled float32 linear projection maps the whitened 128-channel DINAC
latent directly to RGB at one eighth of the requested image size. Every solver
update is eligible by default. If the previous preview is still transferring,
converting, or running the callback, the new update is dropped instead of
blocking sampling. The callback receives this native one-eighth-size image;
the web UI lets the browser scale it for display.

Accepted previews perform only a 1×1 projection and pixel shuffle on CUDA.
The small RGB result is copied non-blockingly into pinned host memory. CUDA
event waiting, PIL conversion, and the application callback all run on a
single background worker. Every step is eligible; the one-worker busy-drop
policy supplies backpressure without a separate callback cap.

## Inference configuration

### Defaults

```python
from canter import (
    CanterInferenceConfig,
    CfgGuidance,
    PdgCurve,
    PdgGuidance,
    PdgMode,
    Schedule,
    Solver,
)

config = CanterInferenceConfig(
    height=1216,
    width=832,
    steps=50,
    solver=Solver.ABM2,
    schedule=Schedule.BETA,
    log_snr_shift=0.0,
    cfg=CfgGuidance(
        enabled=False,
        scale=None,
        start_step=0,
        stop_step=None,
    ),
    pdg=PdgGuidance(
        enabled=True,
        mode=PdgMode.FULL,
        curve=PdgCurve.CONSTANT,
        noisy_scale=2.5,
        clean_scale=2.5,
        power=3.0,
        start_step=0,
        stop_step=None,
    ),
    self_attention_gain=-0.03,
    sde_noise_multiplier=1.0,
    seed=42,
    generator=None,
)
```

### `CanterInferenceConfig`

| Field | Default | Description |
| --- | --- | --- |
| `height` | `1216` | Output height in pixels. Must be positive and divisible by 16. |
| `width` | `832` | Output width in pixels. Must be positive and divisible by 16. |
| `steps` | `50` | Number of solver state updates. A run with 50 updates uses 51 schedule points. |
| `solver` | `Solver.ABM2` | Numerical solver. |
| `schedule` | `Schedule.BETA` | Timestep spacing. |
| `log_snr_shift` | `0.0` | Additive log-SNR schedule shift. Zero leaves the selected schedule unchanged. |
| `cfg` | disabled | Classifier-free guidance settings. |
| `pdg` | full-path PDG 2.5 | Path-drop guidance settings. |
| `self_attention_gain` | `-0.03` | Gain applied exclusively to image self-attention on the main denoiser path. |
| `sde_noise_multiplier` | `1.0` | Shared non-negative stochastic noise multiplier used by Euler-Maruyama and ER-SDE. |
| `seed` | `42` | Random seed. Set to `None` when supplying `generator`. |
| `generator` | `None` | Optional CUDA `torch.Generator` on the same device as the model. Exactly one of `seed` and `generator` is required. |

The seed controls latent noise, SPRINT routing, and stochastic solver
operations. The VAE decoder retains its published seed behavior.

### PNG metadata

Images downloaded from the Gradio interface contain a `canter` PNG text field
with compact JSON. The object begins with the prompt, optional negative prompt,
effective per-image seed, width, height, steps, solver, and schedule. It then
records PDG, CFG,
self-attention gain, logSNR shift, the Euler-Maruyama and ER-SDE noise
multipliers, installed code version, numbered checkpoint release, and weight
dtype.

### Solvers

| Enum | Value | Description |
| --- | --- | --- |
| `Solver.EULER` | `"euler"` | First-order deterministic Euler updates. |
| `Solver.EULER_MARUYAMA` | `"euler_maruyama"` | Stochastic reverse-SDE updates. Uses `sde_noise_multiplier`. |
| `Solver.ER_SDE` | `"er_sde"` | Third-stage VP ER-SDE with 16-point Gauss-Legendre correction quadrature. Uses `sde_noise_multiplier`. |
| `Solver.DPMPP_2M` | `"dpmpp_2m"` | Flow-matching DPM++ 2M updates with finite pre-shift endpoint handling. |
| `Solver.ABM2` | `"abm2"` | Variable-step Adams-Bashforth-Moulton updates with corrected-state reevaluation. |

ABM2 is the default and performs additional denoiser evaluations for its
corrected states.

### Schedules

| Enum | Value | Description |
| --- | --- | --- |
| `Schedule.LINEAR` | `"linear"` | Uniform spacing from noisy to clean. |
| `Schedule.BETA` | `"beta"` | Beta(0.6, 0.6) quantile spacing with more schedule density near the endpoints. |

Schedules and solver state remain float32.

In the bundled web interface, selecting Beta(0.6, 0.6) resets the log-SNR shift
to `0.0`; selecting Linear resets it to `-2.3`. The shift remains independently
editable after either preset is applied. Direct API configurations use the
explicit `log_snr_shift` supplied by the caller.

## CFG

```python
from canter import (
    CanterInferenceConfig,
    CanterPipelineConfig,
    CfgGuidance,
    PdgGuidance,
    PdgMode,
)

cfg = CfgGuidance(
    enabled=True,
    scale=3.0,
    start_step=0,
    stop_step=29,
)

output = pipe(
    "A studio portrait with soft natural light",
    negative_prompts="oversaturated, harsh contrast",
    config=CanterPipelineConfig(
        inference=CanterInferenceConfig(
            cfg=cfg,
            pdg=PdgGuidance(mode=PdgMode.COMBINED_CFG_PDG),
        ),
    ),
)
```

### `CfgGuidance`

| Field | Default | Description |
| --- | --- | --- |
| `enabled` | `False` | Enable classifier-free guidance. |
| `scale` | `None` | Non-negative guidance scale. Required when CFG is enabled or when the selected PDG mode uses CFG. |
| `start_step` | `0` | First active solver update, inclusive. |
| `stop_step` | `None` | Last active solver update, inclusive. `None` selects the final update. |

Step indices run from `0` through `steps - 1`.

An explicit negative prompt replaces the learned unconditional token on CFG
and contrastive PDG branches. Negative prompts require enabled CFG or a PDG
mode that consumes contrastive text. For prompt batches, supply one negative
prompt per positive prompt.

## PDG

```python
from canter import PdgCurve, PdgGuidance, PdgMode

pdg = PdgGuidance(
    enabled=True,
    mode=PdgMode.FULL,
    curve=PdgCurve.POWER,
    noisy_scale=2.0,
    clean_scale=2.5,
    power=3.0,
    start_step=0,
    stop_step=None,
)
```

### `PdgGuidance`

| Field | Default | Description |
| --- | --- | --- |
| `enabled` | `True` | Enable path-drop guidance. |
| `mode` | `PdgMode.FULL` | Path and CFG interaction policy. |
| `curve` | `PdgCurve.CONSTANT` | Scale interpolation from noisy to clean. |
| `noisy_scale` | `2.5` | PDG scale at the first noisy schedule point. |
| `clean_scale` | `2.5` | PDG scale at the final clean schedule point. |
| `power` | `3.0` | Positive exponent used by the power curve. |
| `start_step` | `0` | First active solver update, inclusive. |
| `stop_step` | `None` | Last active solver update, inclusive. `None` selects the final update. |

Constant PDG requires equal `noisy_scale` and `clean_scale`. Disabled PDG
requires `enabled=False` and `mode=PdgMode.NONE`.

Increasing the PDG scale sharpens the generated distribution. Moderate values
improve image structure and fine detail at the cost of reduced variety.
Pushing the scale too far can introduce structural defects, excessive
contrast, and oversaturation.

### PDG curves

| Enum | Value | Scale behavior |
| --- | --- | --- |
| `PdgCurve.CONSTANT` | `"constant"` | Uses one scale throughout inference. |
| `PdgCurve.LINEAR` | `"linear"` | Interpolates linearly from `noisy_scale` to `clean_scale`. |
| `PdgCurve.POWER` | `"power"` | Interpolates with `position ** power`. |

### PDG modes

| Enum | Value | Behavior |
| --- | --- | --- |
| `PdgMode.NONE` | `"none"` | No PDG branch. Required when PDG is disabled. |
| `PdgMode.FULL` | `"full"` | Guides from the middle-skipped path toward the full main path. |
| `PdgMode.FULL_CONTRASTIVE` | `"full_contrastive"` | Uses negative text on the middle-skipped path, or learned unconditional text when no negative prompt is supplied. |
| `PdgMode.THREE_QUARTER` | `"three_quarter"` | Guides from the 75 percent SPRINT path toward the full main path. |
| `PdgMode.ALTERNATE_PDG_FIRST` | `"alternate_pdg_first"` | Alternates PDG on even updates and CFG on odd updates within the PDG window. |
| `PdgMode.ALTERNATE_CFG_FIRST` | `"alternate_cfg_first"` | Alternates CFG on even updates and PDG on odd updates within the PDG window. |
| `PdgMode.COMBINED_CFG_PDG` | `"combined_cfg_pdg"` | Evaluates CFG and PDG together and averages their guidance deltas. |
| `PdgMode.PDG_WITH_ALTERNATING_CFG` | `"pdg_with_alternating_cfg"` | Applies PDG on every active update and adds CFG on odd updates. |
| `PdgMode.CFG_TO_PDG` | `"cfg_to_pdg"` | Uses CFG before the PDG window, then uses full-path PDG inside the window. |

The five compound modes require `cfg.scale`. `PdgMode.FULL_CONTRASTIVE` uses
the PDG scale only. `start_step=0` with `PdgMode.CFG_TO_PDG` begins directly
with PDG.

`pdg_branch_conditioning(mode)` returns `PdgBranchConditioning.POSITIVE` or
`PdgBranchConditioning.CONTRASTIVE` for the selected PDG alternative path.

## Self-attention gain

`self_attention_gain` changes image self-attention only on the full main path.
The model multiplies main-path attention queries by
`exp(self_attention_gain)`. Negative values reduce the attention-logit scale
and therefore increase the effective softmax temperature. This softens the
main prediction and helps moderate PDG oversaturation. Text self-attention,
cross-attention, and weak guidance paths retain their trained scales.

The default is `-0.03`. A value of `0.0` uses the trained main-path
self-attention scale without adjustment. Smaller images generally benefit from
more negative values. As image size increases, the gain should move closer to
zero.

## Custom configuration example

```python
from canter import (
    CanterInferenceConfig,
    CanterOutputType,
    CanterPipelineConfig,
    CfgGuidance,
    PdgCurve,
    PdgGuidance,
    PdgMode,
    Schedule,
    Solver,
)

inference = CanterInferenceConfig(
    height=1024,
    width=1024,
    steps=40,
    solver=Solver.DPMPP_2M,
    schedule=Schedule.LINEAR,
    log_snr_shift=0.5,
    cfg=CfgGuidance(
        enabled=True,
        scale=3.0,
        start_step=0,
        stop_step=11,
    ),
    pdg=PdgGuidance(
        enabled=True,
        mode=PdgMode.CFG_TO_PDG,
        curve=PdgCurve.POWER,
        noisy_scale=2.0,
        clean_scale=2.5,
        power=3.0,
        start_step=12,
        stop_step=39,
    ),
    self_attention_gain=-0.03,
    sde_noise_multiplier=1.0,
    seed=123,
    generator=None,
)

config = CanterPipelineConfig(
    inference=inference,
    output_type=CanterOutputType.PIL,
)
image = pipe("A glass greenhouse during heavy rain", config=config).image
```

## Prompt batches

A sequence of prompts generates one image per prompt:

```python
output = pipe(
    [
        "A windswept beach under dark clouds",
        "A sunlit kitchen with white tiled walls",
    ]
)

first, second = output.images
```

Batch size one is the primary inference path. The jagged backend packs active
prompt tokens without padding them through text refinement and
cross-attention. Prompts longer than 512 tokens are truncated with a warning.

## Latent generation

Use `CanterInferenceEngine` to generate latents without loading or calling the
VAE:

```python
from canter import CanterComponents, CanterInferenceEngine

components = CanterComponents.from_pretrained("data-archetype/canter")
engine = CanterInferenceEngine(components)
output = engine.generate("A mountain road in winter")

latents = output.latents
schedule = output.schedule
```

`CanterInferenceEngine.generate` accepts:

| Parameter | Default | Description |
| --- | --- | --- |
| `prompts` | required | One string or a sequence of strings. |
| `negative_prompts` | `None` | Optional negative prompt string or sequence for CFG and contrastive PDG branches. The count must match `prompts`; blank text uses learned unconditional conditioning. |
| `config` | `CanterInferenceConfig()` | Latent inference settings. |
| `initial_noise` | `None` | Optional float32 noise with shape `[batch, 128, height / 16, width / 16]`. |
| `progress` | `None` | Optional callback receiving `(completed_updates, total_updates)`. |
| `state_callback` | `None` | Optional synchronous callback receiving `(state, completed_updates, total_updates)` after every solver update. Prefer pipeline previews when full CUDA latent access is not required. |

An empty or whitespace-only prompt uses the learned unconditional token without
running the text encoder. Every prompt in a batch must be either blank or
nonblank.

The returned latents use float32 and channels-last memory format.

## Direct VAE encoding and decoding

`CanterVae` exposes both directions of the pinned DINAC-AE latent interface:

```python
import torch
from canter import CanterVae

vae = CanterVae.from_pretrained(
    "data-archetype/dinac_ae_d2",
    device="cuda",
)
images = torch.zeros((1, 3, 1216, 832), device="cuda", dtype=torch.float32)
latents = vae.encode(images)
reconstructed = vae.decode(latents, height=1216, width=832)
```

Encoder inputs and decoder outputs are float32 BCHW tensors in the DINAC image
range `[-1, 1]`. Encoded latents are whitened float32 tensors with 128 channels
and a spatial stride of 16.

## Conditioning, guidance, and solver APIs

The public inference API exposes text conditioning, guidance, schedules,
solvers, and the latent-RGB preview projection as separate components:

```python
import torch
from canter import (
    LOGSNR_SOLVER_START_EPS,
    CanterGuidanceConfig,
    CanterGuidedVelocity,
    LatentRgbProjection,
    Schedule,
    Solver,
    build_schedule,
    solve,
)

conditional = engine.prepare_conditioning("A mountain road in winter")
unconditional = engine.prepare_unconditional(batch=1)
guidance = CanterGuidanceConfig()
generator = torch.Generator(device=engine.device).manual_seed(42)
velocity = CanterGuidedVelocity(
    components.model,
    conditional,
    unconditional,
    guidance,
    generator,
)
schedule = build_schedule(
    Schedule.BETA,
    steps=guidance.steps,
    log_snr_shift=0.0,
    finite_noisy_endpoint_epsilon=None,
    device=engine.device,
)
latents = solve(
    velocity,
    initial_noise,
    schedule,
    solver=Solver.ABM2,
    generator=generator,
)
preview_projection = LatentRgbProjection.bundled()
```

`prepare_conditioning` validates a string or prompt sequence and performs the
bundled text-encoder/refinement pass. An entirely blank batch returns learned
unconditional conditioning; mixed blank and nonblank batches are rejected.
`prepare_unconditional` creates learned conditioning for an explicit positive
batch size.

`CanterGuidanceConfig` contains the solver-step guidance state: step count,
CFG, PDG, and self-attention gain. `CanterGuidedVelocity`, `build_schedule`,
and `solve` preserve the same PDG, CFG, self-attention gain, stochastic
generator, and float32 solver behavior as `CanterInferenceEngine.generate`.

For `Solver.DPMPP_2M` and `Solver.ER_SDE`, pass
`LOGSNR_SOLVER_START_EPS` as `finite_noisy_endpoint_epsilon`; other solvers use
`None`. `LatentRgbProjection.bundled()` returns the validated CPU float32 1×1
projection weight and bias used by native previews.

## Pipeline metadata

`pipe.metadata` records the installed Canter code version, resolved checkpoint
release, weight dtype, source digests, text-encoder revision, VAE repository,
and resolved immutable VAE revision. Applications that require reproducibility
should store this metadata and pin both the package version and checkpoint tag.