hancheolp/test-4bit50

Qwen3.8-27B MLX 4-bit with 50% of FFN at 2-bit

One rung of a five-model ladder built to measure decode throughput vs. FFN bit-width on Apple Silicon. Accuracy was deliberately not tuned — these exist to answer one question:

Is MLX's 2-bit qmv kernel as efficient as the 4-bit one?

This model

Base scheme is uniform 4-bit, group_size 64, affine, matching mlx-community/Qwen3.8-27B-4bit (498 quantized modules, vision tower bf16, MTP dropped). On top of that, 96 of the 192 FFN tensors are dropped to 2-bit (still g64/affine). Attention, lm_head and embed_tokens stay at 4-bit.

• Layers with all three FFN projections at 2-bit: 26 (L31-63)
• Layers only partially converted: 12

2-bit FFN tensors	96 / 192
On disk	13.92 GB
Weights read per decoded token	12.27 GB
Bandwidth-bound ceiling vs. baseline	1.174x

That last number is arithmetic, not a measurement. It is baseline_bytes / this_model_bytes, assuming batch-1 decode is purely memory-bandwidth bound. embed_tokens is excluded from the read figure because decoding gathers a single row rather than streaming the matrix. No tok/s has been measured on any hardware. Real measurements, when they exist, belong below this line.

The full ladder

model	2-bit FFN tensors	disk	ceiling
test-4bit	0 / 192	16.05 GB	1.000x
test-4bit25	48 / 192	14.98 GB	1.080x
test-4bit50	96 / 192	13.92 GB	1.174x
test-4bit75	144 / 192	12.85 GB	1.286x
test-4bit100	192 / 192	11.78 GB	1.422x

Even at 100% FFN coverage the ceiling is 1.42x, and quantizing everything to 2-bit would only reach 1.80x. The g64 metadata (fp16 scale + fp16 bias = 0.5 bpw) does not shrink with bit-width, so 4-bit is really 4.5 bpw and 2-bit is 2.5 bpw.

Which tensors go to 2-bit

Selected by ascending KL sensitivity, using the per-tensor sweep published in mlx-community/Qwen3.8-27B-OptiQ-4bit (optiq/sensitivity.json). FFN sensitivity in this model falls monotonically with depth — mean KL is 0.01584 for L0-7 and 0.00072 for L56-63, a 22x spread — so the least-sensitive tensors all sit near the output. All three FFN projections hold the same parameter count, so coverage alone fixes size and speed; the ranking only decides which tensors take the damage.

Build

Quantized with AutoRound 0.15.0 in plain RTN mode (iters=0, disable_opt_rtn=True, data-free), exported via --format mlx, then repaired.

The repair step is not optional. AutoRound's MLX exporter leaves 97 layers unquantized:

• embed_tokens — SUPPORTED_LAYER_TYPES is (Linear, Conv2d, Conv1D); nn.Embedding entries are dropped by the layer-config resolver.
• linear_attn.in_proj_a / in_proj_b (96 tensors, shape [48, 5120]) — _is_mlx_quantizable() requires out_dim % 64 == 0. MLX imposes no such rule, and mlx-community/Qwen3.8-27B-4bit quantizes all 96.

Those 97 are filled in afterwards with mx.quantize at 4-bit/g64, so the packing is bit-exact MLX rather than a reimplementation of the affine formula. The exporter also omits "mode": "affine" and emits ~57 stray false entries for vision layers; both are fixed.

Usage

from mlx_vlm import load, generate
model, processor = load("hancheolp/test-4bit50")

Caveats

• Accuracy is unmeasured. This rung is not recommended for real use.
• Plain RTN, no calibration. Not representative of AutoRound's tuned modes.
• MTP is dropped, as in the mlx-community conversion. For speculative decoding see mlx-community/Qwen3.8-27B-MTP-4bit.