The column you dropped from the post is the one that tells you how much to trust the other one.

Your repo ships data/results.tsv, and it has a fifth column the post does not: pp512_tps. It also ships bench/run_bench.sh and data/wiki.test.raw. Corpus plus harness plus raw numbers is rarer than it should be, so I ran the table against itself instead of guessing.

Your two imatrix rows are a control, and I do not think you set them up as one.

From the repo tree, exact bytes:

Qwen3-4B-Q4_K_M.gguf          2,497,280,288
Qwen3-4B-Q4_K_M_imatrix.gguf  2,497,280,608     +320 bytes
Qwen3-4B-Q3_K_M.gguf          2,075,617,568
Qwen3-4B-Q3_K_M_imatrix.gguf  2,075,617,888     +320 bytes

320 bytes on 2.5 GB. Same quant type, same tensor layout, same working set. imatrix picks which weights get the precision at quantize time, it does not change what the CPU does at generate time. So for throughput each pair is the same computation run twice, and the gap between them is your noise floor:

                 tg128 t/s        pp512 t/s
Q4 pair       6.50 vs 6.41  1.4%   2064.82 vs 1965.28  4.8%
Q3 pair       6.56 vs 5.89 10.2%   2142.51 vs 2308.21  7.7%

Three of the speed comparisons in the post are smaller than that.

Q4_K_M 6.50 vs Q3_K_M 6.56          0.9%   (11x below the Q3 pair's own spread)
Q6_K   5.93 vs Q5_K_M 5.64          4.9%   and Q6_K is the BIGGER file

Q5_K_M coming out slower than a file 416 MB larger is the tell. Going down the ladder should buy speed monotonically on a memory-bound CPU run, and between Q6 and Q3 it does not, because the differences there are inside the run-to-run spread your own control pair measures.

None of this touches your recommendation.

Perplexity has no run-to-run noise here. Fixed corpus, fixed ctx 2048, -s 1, deterministic given the weights. And the effect is enormous next to anything above:

F16 13.4304 -> Q3_K_M 15.6641    +2.2337   16.6%
imatrix at Q3  15.6641 -> 14.8237  -0.8404   5.4%
imatrix at Q4  13.7304 -> 13.6760  -0.0544   0.40%

"Q4 if you have the memory, imatrix Q3 if you don't" is carried entirely by that column, and it holds. The t/s column just cannot rank Q4 against Q3, so the size argument is doing the work there anyway.

The number that settles it is already on your disk.

run_bench.sh runs llama-bench -r 3 and tees bench/bench_$q.md, which is where llama-bench prints the standard deviation. Then:

tg128=$(awk -F'|' '/tg128/{gsub(/[^0-9.]/,"",$8); print $8}' "bench/bench_$q.md" | head -1)

strips every non-digit out of the t/s cell, so whatever came after the ± does not reach results.tsv in any form. And the repo has the GGUFs, the harness, the corpus and the TSV, but not the 16 bench_*.md / perp_*.log files the script writes. They are the only outputs that did not get uploaded, and they are a few KB.

What was the ± on tg128 for Q3_K_M?

You shipped the files, so the answer is better than the one number.

±0.27 is right for Q3_K_M. It is not the noise floor, because there isn't one. Across the eight bench_*_v2.md the tg128 sd spans 7x:

Q4_K_M          6.50 ± 0.15    2.3%
Q6_K            5.93 ± 0.19    3.2%
Q3_K_M          6.56 ± 0.27    4.1%
Q4_K_M_imatrix  6.41 ± 0.27    4.2%
Q8_0            5.04 ± 0.39    7.7%
Q5_K_M          5.64 ± 0.96   17.0%
Q3_K_M_imatrix  5.89 ± 1.05   17.8%
F16             2.52 ± 0.98   38.9%

Two rows in the table go the wrong way, a smaller file benchmarking slower than the one above it:

Q6_K   3.31 GB  5.93   ->  Q5_K_M          2.89 GB  5.64
Q4_K_M 2.50 GB  6.50   ->  Q3_K_M_imatrix  2.08 GB  5.89

Those are the two highest-sd rows in the set. Next worst is 7.7%. Both anomalies sit on the two noisiest measurements and nothing else in the column is out of order.

Stepping down the size ladder, only the top two steps separate at n=3:

F16 -> Q8_0       +2.52   resolves
Q8_0 -> Q6_K      +0.89   resolves
Q6_K and below     5 steps, none resolve

Same verdict whether the ± is a sd or a sem. Both quants you recommend live in the part of that column that cannot be ranked.

One thing worth fixing before the next run.

run_bench.sh reads t/s as field 8. In these v2 outputs field 8 is the test name, because the CUDA build prints an ngl column the earlier one did not. Your line, against your own file:

awk -F'|' '/pp512/{gsub(/[^0-9.]/,"",$8); print $8}'  ->  512
awk -F'|' '/tg128/{gsub(/[^0-9.]/,"",$8); print $8}'  ->  128

$9 does not save it either, because the gsub that strips the ± welds the sd onto the mean: 6.56 ± 0.27 becomes 6.560.27.

Field-count-proof, since the row always ends on a bar:

{ v = $(NF-1); gsub(/^[ \t]+|[ \t]+$/,"",v); split(v, a, /[ \t]*±[ \t]*/); print a[1] "\t" a[2] }

That reproduces all 16 of your published means exactly and gets the 16 sds for free. Separately, the script writes bench/results.tsv but the table in the repo is data/results.tsv. Which one is canonical?

Perplexity is where the recommendation actually lives, and it holds.

Q3 imatrix  15.6641 -> 14.8237   -0.8404   5.9x the log's own ±0.14
Q4 imatrix  13.7304 -> 13.6760   -0.0544   0.45x of ±0.12

The per-arm ± is 0.873% to 0.878% of PPL on six of the eight, so it is mostly corpus variance every arm shares, and that cancels in a paired comparison. You just cannot run the paired comparison from a log that prints only the total.

llama-perplexity --kl-divergence-base on the F16, then --kl-divergence on each quant, is built for that exact gap. Would it change where you draw the Q4 line?

Your new extractor reproduces all 16 published means, and the path you meant to fix is still pointing at bench/.

Ran the 5c26bde version against your own eight _v2.md files:

pp512 and tg128 means vs data/results.tsv    16 of 16 exact, 0 mismatches

Capturing the sds too was the better move than what I suggested, and it changes what the table can say. Here is the column you now get for free:

                 pp512        ±      rel     tg128      ±     rel
F16             357.63   216.44   60.5%       2.52   0.98   38.9%
Q8_0           1355.89    30.55    2.3%       5.04   0.39    7.7%
Q6_K           1523.24    86.17    5.7%       5.93   0.19    3.2%
Q5_K_M         1917.10   154.18    8.0%       5.64   0.96   17.0%
Q4_K_M         2064.82   152.87    7.4%       6.50   0.15    2.3%
Q4_K_M_imx     1965.28   116.49    5.9%       6.41   0.27    4.2%
Q3_K_M         2142.51   228.68   10.7%       6.56   0.27    4.1%
Q3_K_M_imx     2308.21   171.38    7.4%       5.89   1.05   17.8%

F16 is the noisiest row on both columns, 60.5% on pp512 against 10.7% for the next worst. It is also the only 8 GB file in a low-RAM test, so that row measures the pager at least as much as the model. Anything quoted as a percentage of F16 inherits it.

Two small things from the push.

Line 12 is still OUT="bench/results.tsv", so the script writes where you said it should not.

The new header is 7 columns and the committed data/results.tsv is still 5. The next run will not drop into the existing table, so regenerate it rather than pasting the two sd fields in by hand.

Would you publish the sd columns before a re-run? The two rows where a smaller file benchmarks slower are the two highest-sd rows in the tg128 column, 17.8% and 17.0%, and right now a reader cannot see that from the table.

Both fixes landed. The same bug has a third instance, and it is sitting on the perplexity column.

Pulled b81a10f1. Line 12 is OUT="data/results.tsv" now, and the committed table is 7 columns. I re-extracted all 32 numbers from your eight _v2.md files with your own lines 26-31: 16 means and 16 sds, zero mismatches against the published table. That part is clean.

Line 20 is not:

ppl=$("$P" ... 2>"bench/perp_$q.log" | grep -oE 'PPL = [0-9.]+' | ...)

2> sends stderr to the log. The pipe carries stdout. And the line you are grepping for is inside the file you redirected away:

perp_F16.log:  ... I Final estimate: PPL = 13.4304 +/- 0.11739

One hit in every one of the eight logs, zero available to the pipe. I ran your exact line against a stub that prints that on stderr:

captured ppl=[]   len 0
row appended:   F16 <TAB> 8.051 <TAB> <TAB> 357.63 <TAB> 216.44 <TAB> 2.52 <TAB> 0.98

Seven fields, third one empty. 2>&1 >/dev/null | tee "bench/perp_$q.log" gives back 13.4304.

Same shape as the OUT bug. The script writes the evidence to one place and reads from another.

Two more from the same pass.

Line 24 tees to bench/bench_$q.md, but the published files are bench_$q_v2.md. A re-run writes eight new files next to the old ones and leaves the published raw logs untouched. The table moves, the evidence does not.

And data/results.tsv is CRLF while run_bench.sh is LF. Bash echo -e does not emit CRLF, so the committed table did not come out of a plain run of this script. Which is consistent with the paragraph above.

That last one is falsifiable, so use it: re-run as-is and results.tsv should come back LF-only with a blank perplexity column. If it does not, I have one of these wrong.

One correction to my own last comment while I am here. I wrote that Q5_K_M and Q3_K_M_imatrix are "the two highest-sd rows in the tg128 column" and you repeated it. They are not. 17.0% and 17.8%, against F16 at 38.9%. F16 is the noisiest row in that column by a factor of two, and I had said so myself four paragraphs earlier.

The +/- is on the same log line as the PPL, and you are dropping it:

Q3_K_M          15.6641 +/- 0.14163
Q3_K_M_imatrix  14.8237 +/- 0.13169

Would you capture it in the same fix? It is a ceiling, not the paired test, but a reader can at least see that the imatrix delta is six times it while the Q4 one is half.