EximiusLabs/fusion-embedding-2-ember

Ember — the thermal sense for fusion-embedding-2

Ember is the first sense pack for fusion-embedding-2: it teaches the

model to embed thermal infrared images in the same vector space as its text,

image, video, and audio embeddings. Packs are named for the physical trace their

sensor reads; Ember reads heat.

Ember is strictly additive. Technically it is a 44M-parameter gated adapter pack

that attaches to the frozen decoder behind a thermal-only gate: when the gate is

closed (every non-thermal input), the model's outputs are bit-for-bit identical

to the model without the pack. This is verified, not aspirational; see

Correctness below.

What it does

• Thermal image to text retrieval: R@10 0.785 on a held-out 2,000-caption gallery

  (frozen base: 0.224).

• Thermal zero-shot classification is preserved: LLVIP person/background 94.3

  (calibrated ensemble harness; frozen base reference 95.4).

• Cross-domain thermal-to-visible retrieval: R@10 0.348 on LLVIP registered pairs,

  above the frozen baseline 0.165, without training on any LLVIP data.

• Text, RGB image, video, and audio embeddings unchanged, bit-for-bit.

Usage

Ember loads as an adapter pack through the multi-gate adapter registry in the

fusion-embedding GitHub repository

(fusion_embedding/adapters.py). Thermal images are single-channel; replicate to

three channels and encode through the ordinary image path with the thermal scope open.

import torch

from huggingface_hub import hf_hub_download

from safetensors.torch import load_file

from fusion_embedding.adapters import AdapterPacks

from inference import FusionEmbedder          # fe2_release/inference.py (GitHub repo)



emb = FusionEmbedder.from_pretrained(

    "EximiusLabs/fusion-embedding-2-2b-preview", revision="v0.2-preview", device="cuda")



packs = AdapterPacks()

adapters, gate = packs.add_pack("thermal", emb.model.base_lm, 2048, rank=384)

adapters.load_state_dict(load_file(hf_hub_download(

    "EximiusLabs/fusion-embedding-2-ember", "model.safetensors")))

packs.to("cuda")



# thermal encode: thermal readout template, thermal scope open (the scope

# spans forward and backward)

import torch.nn.functional as F

text = ("<|im_start|>system\nRepresent this thermal infrared image.<|im_end|>\n"

        "<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|><|im_end|>\n"

        "<|im_start|>assistant\n")

inp = emb.proc(text=[text], images=[thermal_image_3ch], return_tensors="pt").to("cuda")

with torch.no_grad(), packs.scope("thermal"):

    h = emb.full(**inp).last_hidden_state

thermal_vec = F.normalize(h[0, inp["attention_mask"][0].sum() - 1].float(), dim=-1)



# everything else: leave the scope closed; outputs equal the pack-free model exactly

text_vec = emb.embed_text("a person crossing a dark road")

audio_vec = emb.embed_audio(wav, sr=16000)    # audio pack co-loaded, unaffected

The pack attaches equally to the raw base (AutoModel.from_pretrained("Qwen/Qwen3-VL-Embedding-2B"),

attach at .language_model), which is the exact configuration it was trained in;

both loading paths are verified bitwise in the release smoke. The readout is the

standard fusion-embedding protocol: chat template, last non-pad token pooling,

L2 normalization (see config.json for the exact templates and the trained

temperature).

Evaluation

Training: contrastive thermal-to-caption alignment on IR-TD (61,320 pairs after

FLIR exclusion and eval dedup), 3,900 steps, batch 16, 1,024 bank negatives,

frozen base, bf16 base precision with fp32 adapters. Three seeds; seed 2 shipped.

| release run (61K corpus) | holdout t2t R@10 | delta vs frozen | LLVIP-ZS | LLVIP twin R@10 |

|---|---|---|---|---|

| frozen base | 0.224 | - | 95.4 | 0.165 |

| seed 1 | 0.783 | +0.560 | 94.1 | 0.333 |

| seed 2 (shipped) | 0.785 | +0.561 | 94.3 | 0.348 |

| seed 3 | 0.777 | +0.554 | 91.8 | 0.341 |

Seed 2 holdout detail: R@1 0.412, R@5 0.692, R@10 0.785 over a 2,000-item gallery.

Text to thermal retrieval on the release holdout (queries shortened for display;

retrieval used the full captions):

A caption-style ablation on the pre-exclusion corpus (82K pairs, 5,000 steps)

found that caption richness is a generalization lever, not just an in-domain fit

lever: training on full descriptive captions reached holdout R@10 0.843 and LLVIP

twin 0.343, while first-sentence captions reached 0.594 and collapsed cross-domain

transfer to 0.089, below the frozen baseline. Ember ships the full-caption arm.

Domain note: IR-TD spans 63 source collections but is still a finite domain mix.

The LLVIP numbers above are cross-domain signal (night pedestrian scenes never

seen in training), not a claim of parity with in-domain retrieval. Expect the gap

to vary with distance from the training domains.

Correctness

The bit-for-bit preservation claim is tested at three levels:

1. Unit suite (GitHub repo, tests/test_thermal_adapters.py): closed-gate

   forwards equal the base exactly; gradients reach only the open pack; the gate

   must span forward and backward under gradient checkpointing.

2. Release checks with the trained weights: RGB-image and text forwards

   bit-for-bit equal to the pack-free base with the thermal gate closed, per seed.

3. Composability matrix (audio pack + thermal pack co-loaded on the same frozen

   decoder): audio through the registry vs the shipped single-gate path, audio

   co-loaded vs audio-only, thermal co-loaded vs thermal-only, and text / RGB /

   video vs the raw base, all bitwise; retrieval scores identical under co-load.

Mixed inputs that would open two gates in a single forward are outside the

guarantee and are not tested.

Provenance

• Training corpus: IR-TD early access (IRGPT, ICCV 2025,

  arXiv:2507.14449,

  repository); 84,284 real thermal

  images with LLM-generated descriptive captions; academic research use only.

• FLIR exclusion: IR-TD includes FLIR-derived sources whose terms restrict

  redistribution of trained weights. All 20,964 images matching the FLIR capture

  signature (640x512) were excluded from training and holdout. This is a

  size-based heuristic, not an author-provided source mapping; the exclusion list

  ships in this repository (release_strip_640x512.json, sha256 65a870df5e4fe0008fd9bacd9fa81bd0c47f919779a28e398fea8d1bcabad09f).

• LLVIP is used for evaluation only (zero-shot gate and cross-domain retrieval).

• Eval hygiene: perceptual-hash dedup between the training set and the LLVIP test

  set found 0 collisions (hamming distance <= 4).

Deploy on RunPod

One-click deploy the endpoint from the RunPod Hub (serverless, scales to zero when idle). Once it is running, call it:

curl -s https://api.runpod.ai/v2/<ENDPOINT_ID>/runsync   -H "Authorization: Bearer $RUNPOD_API_KEY"   -H "Content-Type: application/json"   -d '{"input": {"thermal": "<https url | data-uri | base64>"}}'

Use text instead of thermal to embed a caption. Returns 2048-d vectors, so thermal and text land in one space.

License

The Ember weights in this repository are released under CC-BY-NC-4.0 for research

use, reflecting the academic-use terms of the training corpus. The core

fusion-embedding-2 model is a separate artifact under its own license; this pack

is optional and separable, and does not modify the core model's weights.