Seb-9B-v2: another small decision model that also speaks /v1/systemone
For anyone looking for a smaller, edge-friendly option: Disclosure: I'm the author of Seb-9B-v2 (https://huggingface.co/ironbcc/Seb-9B-v2), a 9B decision model on Qwen3.5-9B, Apache-2.0. It returns a probability for every option in one forward pass, speaks /v1/systemone and a plain chat API, reads images, and takes inputs up to 32k tokens. These are self-run numbers, not independent ones: Decision Index 0.3 public index 59.16 (board numbers: Jev 1.13 57.96, Jev 1.14 60.70), ahead of Jev 1.13 on 22 of 37 benchmarks and clearly behind it on knowledge and reasoning. About 22 ms per decision on one RTX PRO 6000, about 88 ms on an Apple M5 Max. Write-up with the misses included: https://huggingface.co/blog/ironbcc/seb-9b-v2
it seems like https://hf.co/Cloudflare/clef-flash would be a better model for this class of hardware, can you elaborate on the benefits Seb 9B v2 has versus our 9B model?
if you could share your datasets publicly, we can also investigate folding any improvements into our next version.
Thanks, fair question. I haven't measured clef-flash on my hardware, so I can't say it's worse for edge. What I can say from the Decision Index public numbers (board for clef-flash, my own run for Seb-9B-v2; self-run, not independent):
| Seb-9B-v2 | clef-flash | |
|---|---|---|
| Public index | 59.16 | 56.15 |
| Language | 62.2 | 47.2 |
| Retrieval | 63.5 | 52.2 |
| Tools | 83.1 | 81.9 |
| Arts | 48.9 | 48.4 |
| Knowledge & reasoning | 39.9 | 53.0 |
Where Seb helps: language understanding, retrieval and classification, and inputs up to 32k tokens (clef-flash defaults to 16k). Where clef-flash is clearly better: knowledge and reasoning (GPQA, MMLU-Pro, BBH), and I haven't compared on the board's Full score (clef-flash is 44.43; I don't know ours). Speed: 22 ms p50 on one RTX PRO 6000 and 88 ms on an M5 Max; your 38.8 ms median is a different setup, so I wouldn't compare them directly.
On datasets, thanks, I'll check what I can release. Some sources have licences that don't allow redistribution, so it may be partial. I'll follow up here.
Adding the full clef family to my earlier reply, since "better or worse" depends on which model. Same caveats: Decision Index 0.3 public index, per benchmark, my own run for Seb-9B-v2 vs the board's numbers for clef (self-run, not independent).
| Seb-9B-v2 (9B) | clef-flash (9B) | clef-omni (30B-A3B) | clef (27B) | |
|---|---|---|---|---|
| Public index | 59.16 | 56.15 | 56.54 | 61.71 |
| Knowledge & reasoning | 39.9 | 53.0 | 53.8 | 52.8 |
| Language | 62.2 | 47.2 | 49.8 | 61.3 |
| Retrieval | 63.5 | 52.2 | 59.3 | 62.8 |
| Tools | 83.1 | 81.9 | 72.6 | 80.7 |
| Arts | 48.9 | 48.4 | 46.4 | 48.9 |
| Benchmarks where Seb is ahead (of 37) | 15 | 15 | 7 | |
| Board Full score | not scored | 44.43 | 45.04 | 54.92 |
Where Seb-9B-v2 is ahead: vs clef-flash, 15 benchmarks, led by POP909 (79.2 vs 0.7), RAGTruth (64.3 vs 0), HoVer (76.2 vs 22.4), iSarcasmEval (45.0 vs 0), VAST (61.2 vs 24.4), When2Call (88.1 vs 53.8) and CLINC150 (93.1 vs 66.7). Vs clef (27B), 7 benchmarks: POP909, HoVer, When2Call, VAST, iSarcasmEval, RAGTruth and ContractNLI (78.0 vs 73.0).
Where the clef models are ahead: knowledge and reasoning (CRUXEval, MuSR, CLadder and WinoGrande are the biggest gaps), and clef (27B) is ahead overall.
The same table is on the model card: https://huggingface.co/ironbcc/Seb-9B-v2
Speed, as each card publishes it (no new runs; the cards don't say what request mix they used):
| Median | p95 | Hardware | |
|---|---|---|---|
| Seb-9B-v2 | 21.7 ms | 36.1 ms | one RTX PRO 6000, 600 W; short rows, mean about 239 tokens |
| clef-flash (9B) | 38.8 ms | 122.4 ms | one H200 |
| clef (27B) | 209.3 ms | 238.6 ms | one H200 |
The published numbers are lower for Seb-9B-v2 even though the H200 is the stronger GPU, but the hardware and the request mix differ, so I'd read it as "lower on each card's own numbers", not "faster". Default context: Seb-9B-v2 32k, clef models 16k.
(Edited to add the speed comparison.)