BAAI/AREX-2 FP8 + DFlash2 on SGLang — TP=2 across 2× GB10

Companion to the Flash-Next card. Same launcher, same fabric, same two boxes;
a different model and a different outcome. AREX-2 is a Qwen3.8-27B fine-tune
with a vision tower and Gated DeltaNet layers, served here at FP8 alongside the
embedding indexer and a co-resident image backend.

The headline is in the Review: it is measurably slower than the base
Qwen3.8-27B-NVFP4 it derives from, and two correctness defects showed up in
long sessions. Both are recorded with sample counts rather than summarised.

---

Hardware

Unified memory means CUDA allocations do not reclaim page cache, and below
roughly 10 GiB free the box freezes rather than OOMs. Everything below is
written around that floor.

---

Model and image

• Target: prithivMLmods/AREX-2-FP8, 67 GB on disk
• Drafter: z-lab/Qwen3.8-27B-DFlash2
• Image: flashnext-sglang:pr45142 (upstream dev + PR #45142)

The drafter was trained against base Qwen3.8-27B, not this fine-tune. It
transfers anyway — see Measured. That result is published nowhere else that we
found.

config.json reports model_type: qwen3_5, architectures: ["Qwen3_5ForConditionalGeneration"], and carries a vision_config. SGLang
instantiates Qwen3_5ForCausalLM from it — the text-only class — which is
worth knowing before reasoning about the vision path.

One trap for anyone loading this model through a tool that builds it with
from_config: the resulting config object has architectures set to None
rather than absent. Anything doing getattr(config, "architectures", []) gets
None back and the default never applies. ModelOpt's is_multimodal_model hits
this and raises TypeError: 'NoneType' object is not iterable; upstream fixed
it with or [].

---

Launch

Through the shared launcher:

./start-cluster.sh arex2

The resolved serve configuration, read off the startup log rather than from the
script:

Three things differ from the base-model profile and are deliberate:

--mamba-ssm-dtype float32. This checkpoint's config asks for float32 where
the other profiles force bfloat16. It doubles GDN state traffic. The bfloat16
comparison has not been run.

8 draft tokens, not 4. Flash-Next caps at 4 because its QSA compress ratio
raises a hard NotImplementedError above it. This model has no such cap.

No --moe-runner-backend flag. Dense checkpoint, so the g1_scale_c
SM12x MoE-runner misroute does not apply. That bug is MoE-path only.

Co-residents

Both are started by the launcher after the LM is serving:

The image backend's installed checkpoints are FLUX.1-schnell —
flux1-schnell-fp8.safetensors (17.2 GB) and flux1-schnell.safetensors
(23.8 GB bf16). Dashboards in this environment label the vision head "FLUX3";
what is on disk is FLUX.1-schnell.

It runs on the worker by design. The head carries LM rank 0 plus the
indexer and reported 13.2 GiB CUDA-visible free with no image model loaded,
against 17–24 GB for a FLUX checkpoint. The worker carries rank 1 only and had
~38 GiB. Running it on the head crosses the freeze floor on the first
generation.

---

Measured

Memory

At --max-mamba-cache-size 120 and mem-fraction-static 0.85, from the
server's own startup log:

The 20.0 KB/token figure was taken twice — once uncapped and once capped — and
agreed to three figures both times. GDN state is reserved per request at a
fixed size regardless of sequence length, which is why slot count caps
concurrency rather than the pool does.

--max-total-tokens is the only lever that returns memory to the system here:
SGLang sizes the KV pool to fill mem-fraction-static, so freeing state yields
a bigger pool rather than free memory, and a cap above what the server would
choose on its own frees nothing.

Speculative decoding

Over 55,796 verify calls:

The base model's drafter transfers to the fine-tune. 3.125 accepted tokens
per verify at 8 drafts, from a drafter trained on Qwen3.8-27B and never
retrained for AREX-2. For comparison, Flash-Next with its own matched drafter
records 2.52 at 4 drafts over 1.39M verify calls.

Accept rate is capped by sampling temperature, not misconfiguration: SGLang's
acceptance thresholds default to 1.0, i.e. exact match, so above temperature 0
a deterministic draft is paid p(argmax) per slot.

Throughput

Peak, over the sample counts shown:

Concurrency curve:

Levels 1 and 2 are measured. Level 3 (n=73) and level 4 (n=2) are
indicative, not measured — the same 100-sample bar the Flash-Next card uses.
Per-stream flattens between 2 and 3 streams rather than continuing to fall,
which is the shape worth re-measuring with real sample counts before relying on
it.

---

Review

Slower than the model it fine-tunes

Against RadixArk/Qwen3.8-27B-NVFP4 on the same two boxes:

Solo decode is the cleanest number on both sides — 3,190 and 2,793 samples —
and AREX-2 gives up 37.6% of it.

The quantization accounts for most of that. AREX-2 is FP8 at ~13.5 GB/node;
the base is NVFP4 at ~6.8 GB/node. Decode on GB10 is memory-bandwidth-bound, so
reading twice the weight bytes per token is a direct, expected cost. It is not
a configuration fault.

Two confounds make this not a pure model comparison. AREX-2 ran with the
indexer on and the vision head on; the base ran with both off. Both
co-residents take memory and some compute on the same nodes. A like-for-like
comparison needs AREX-2 re-measured with EMBED=0 FLUX=0, which has not been
done.

Where AREX-2 holds up better: it degrades less with concurrency —
0.79× vs 0.66× at three streams — and its prefill is within 5%. The gap is
almost entirely in single-stream decode.

An NVFP4 build of AREX-2 could not be made to work

Four configurations were produced and all four served degenerate output: a wall
of token id 0 (! in Qwen's vocabulary), which is argmax over flat or NaN
logits. The same serving config with the FP8 checkpoint is clean, and
DRAFT=none does not change it, so neither the profile nor speculation is
implicated.

What was ruled out along the way, each by measurement:

• Not bad scales. mlp.gate_proj weight, weight_scale, weight_scale_2
  and input_scale were compared tensor-by-tensor against
  RadixArk/Qwen3.8-27B-NVFP4: identical dtypes and shapes, weight_scale_2
  agreeing to four significant figures, no NaNs, no zeros.
• Not 4-bit activations. W4A16 fails identically to W4A4.
• Not the GDN path specifically. The working reference quantizes it —
  linear_attn.in_proj_qkv carries 48 weight_scale tensors there.
• Not metadata. Both checkpoints declare quant_method: modelopt,
  quant_algo: NVFP4, the same config_groups shape and the same ignore key.
• Not an SGLang guard mismatch. qwen3_5.py in this image passes
  quant_config to Qwen3_5GatedDeltaNet unconditionally; there is no
  modelopt_fp4-keyed guard to miss.

The one structural difference not eliminated: the working reference has two
ignore entries (mtp*, mtp.layers.0*) while every build here excluded far
more, and SGLang's packed_modules_mapping fuses
in_proj_qkvz ← [in_proj_qkv, in_proj_z] and in_proj_ba ← [in_proj_b, in_proj_a] at load. The "quantize everything but mtp*" build was the test of
that and also failed, so it remains unexplained. Use FP8.

Correctness defects observed

The long-haul pattern is the more serious of the two, because the usable
artifact is fine while the state behind it is not — a session can appear
healthy from its output alone. Neither defect has a reproducer. Both were seen
with --mamba-ssm-dtype float32, 262144 context and 120 GDN slots; whether
either is implicated is unknown.

---

Relative comparison

Identical section in all four cards, from one snapshot so the numbers agree
across them.

This dataset is being retired. Collection is restarting from cleared
counters, because the configurations behind these samples cannot be
reconstructed. Read the provenance table before any figure below.

Recipe lineage

Original findings in this set, not published elsewhere that we found:

Provenance

Flash-Next is the one profile whose topology is not established. A single-node
deployment of it is possible via file-backed PLE offload — the launcher carries
PLE_OFFLOAD=1 for this model specifically, ~/ple-tables holds 48 GB on the
head, and the stop list names a flashnext-sg container distinct from
flashnext-tp2. Single-node with NVMe in the decode path is slower, so any such
samples drag a cumulative aggregate down.

Known contamination in the Flash-Next series, beyond topology:

Counters are cumulative since process start, so the existing samples cannot be
segmented by configuration after the fact.

What the measurement says should be true and isn't

Flash-Next is a 206-shard MoE. Its decode cost is set by active parameters per
token, a fraction of the total, so it should beat a dense 27B comfortably.
It measures 8% below it. That inversion is the strongest single argument
that the Flash-Next series is not describing one configuration, and it stands
independent of any individual contamination source.

Aggregate throughput, tok/s

No winner is marked. Qwen3.8-27B is the only column with established provenance
and cleared counters, and it is also the thinnest — three levels, two clearing
100 samples.

Per-stream retention — do not rank on this

Every value here is a ratio against the c1 figure, which may be projected
rather than sampled. If it is, these are ratios to a projection and the
ranking they imply — including AREX-2 retaining best — is not established. The
aggregate columns above stand on their own; this table does not.

Cost of the first additional stream

Same c1 dependency as the table above. dsv41's step is more than twice any
other, which is large enough to survive a projected baseline being somewhat off,
but it is not a clean figure either.

Peak against the highest measured aggregate

Sample depth

Depth is not integrity. Flash-Next has the most samples and the least
established provenance.

---

Not measured

Stated plainly because it would otherwise look like data:

• Concurrency above 2 streams. n=73 at three and n=2 at four. The curve
  above two streams is a shape, not a figure.
• The like-for-like comparison against the base model. AREX-2's numbers
  carry a co-resident indexer and image backend; the base model's do not.
• --mamba-ssm-dtype bfloat16. Would halve 22.85 GB of GDN state traffic
  and is the obvious response to the decode gap. Never run; the checkpoint's
  config asks for float32.
• Vision, end to end. The tower loads, but no image request has been
  confirmed against this model with a data-URI payload. models.json still
  advertises "input": ["text"], so the agent harness cannot send one.
• Cold prefill above 32K. Silent worker-rank death near 62K and a
  multi-node wedge at 95–100K are reported for 262K-context configs on this
  hardware. 32K passes here; above that is untested.
• Whether GPU Direct RDMA helps. It is off: nvidia-peermem does not load
  on aarch64 (missing ib_register_peer_memory_client), and DMA-BUF reports
  unavailable even with NCCL_DMABUF_ENABLE=1, NCCL_NET_GDR_LEVEL=SYS and
  NCCL_NET_GDR_READ=1. Collectives stage through host memory. On unified
  memory, host and device are the same physical DRAM, so the usual argument for
  GDR is weaker than on a discrete card — but what it would buy here is
  unmeasured either way.
• MTU persistence. 9000 is set and verified at launch with an 8972-byte
  DF ping, but is not persistent across reboot. No netplan or NetworkManager
  config holds it.

---

Terms, because they are easy to conflate

---

Credits

This profile is original; it inherits its scaffolding:

• The launcher, fabric handling and the six GB10 fixes — flashnext-dflash2-2x-gb10.md
• PR #45142 — DFlash aux hidden-state capture, required by the image
• z-lab/Qwen3.8-27B-DFlash2 — drafter, trained against the base model
• The is_multimodal_model architectures=None fix — upstream ModelOpt
• RadixArk/Qwen3.8-27B-NVFP4 — the working reference checkpoint the failed NVFP4 builds were compared against