SceneWorks/krea2-pose-controlnet-beta

Krea 2 Turbo — Pose ControlNet (S0 SPIKE checkpoints)

⚠️ PURELY EXPERIMENTAL — DO NOT USE IN PRODUCTION. These are feasibility-spike checkpoints from a GO/NO-GO experiment, not a finished model. They were trained on a tiny corpus (5,000 pairs) for a fixed step budget to answer a single yes/no question — can a Krea-native pose control branch be trained at all. They are intermediate artifacts kept only to bootstrap ControlNet interface wiring. The shippable overlay will be a separate, retrained model on the full corpus. Expect artifacts, incomplete pose fidelity, and no stability guarantees.

What this is

A from-scratch pose ControlNet control branch for the Krea 2 Turbo image model (Qwen3‑VL‑4B TE + ~12B 28‑block single‑stream DiT + Qwen‑Image VAE; CFG‑free rectified‑flow, 8‑step). Krea 2 has a bespoke backbone with no transferable ControlNet, so the control branch was trained from scratch. This repo hosts two intermediate checkpoints from the S0 spike:

File	Step	Notes
control_step5000.safetensors (+ .json)	5000	primary — best pose fidelity in the spike
control_step4500.safetensors (+ .json)	4500	fallback (clean pre‑resume checkpoint)

Each .safetensors requires its .json sidecar — it carries the injection metadata (inject_offset, n_blocks, residual‑clamp tau, step) the inference harness reads to configure the branch. Do not use a checkpoint without its sidecar.

Architecture / recipe (spike)

• Branch = a trainable copy of the first N=7 of 28 Krea single‑stream DiT blocks (~3.30B params).
• Reads the VAE‑encoded pose‑skeleton latent; a zero‑initialised output projection per block adds a residual to the frozen main stream (base fully frozen: DiT + TE + VAE).
• inject_offset=1 — branch injects into main block i+1, skipping main block 0.
• Residual RMS clamp τ=0.15 — each injected residual is capped at τ × RMS(main image tokens).
• Loss = Krea flow‑matching over the full noise schedule. Trained bf16 branch, lr 2e‑5, 512², batch 4.

These three choices (skip block 0, clamp, and a low‑lr + weight‑decay optimizer group on the zero‑init projections) are load‑bearing: without them the zero‑init projection grows under AdamW until the branch overwrites the frozen stream and generation collapses to noise.

Intended use

Wiring / integration testing only — standing up and validating a Krea ControlNet inference route (candle/CUDA generate_krea_control_stream, and later the MLX twin). Requires:

• the Krea 2 Turbo base weights (krea/Krea-2-Turbo), and
• the krea-control-infer harness from the candle-gen spike branch (claudeanthropic/sc-8460/krea-2-pose-controlnet-s0-spike).

Recommended control_scale band 0.5–0.75 (default ~0.6–0.7); control_scale = 0 is bit‑exact to the base model. Values ≥ ~0.9 over‑drive the branch and degrade image quality (haze / halftone).

Known limitations

• Body pose‑lock is strong and generalizes to unseen skeletons at 5k pairs; head‑direction is usable but scale‑sensitive (best near control_scale 0.5), not rock‑solid.
• Image quality degrades as control_scale rises — a fundamental authority‑vs‑fidelity tradeoff, sharper here because the base is frozen, CFG‑free, and few‑step. Not fully removable by more data.
• Trained at 512² only. Skeleton convention = OpenPose body‑18 + head‑direction (nose/eyes/ears); no hands, no dense face.

Training data & attribution

Trained on ~5,000 pairs derived from COCO train2017 person_keypoints ground‑truth annotations (skeletons rendered in the controlnet_aux OpenPose drawing convention) with COCO Captions as the text conditioning. COCO annotations are CC BY 4.0; COCO images are Flickr mixed‑license. This checkpoint is a derived model artifact for research/feasibility only.

Provenance

SceneWorks epic 8459 (Krea 2 pose ControlNet), story sc-8460 (S0 spike, verdict = GO). Experimental — retained solely to bootstrap interface wiring; superseded by the production overlay once the full‑corpus run (S3) lands.