Join the community of Machine Learners and AI enthusiasts.
Sign Up
The second-pass arithmetic check is the real finding here, not the token cut. Just thinking longer didn't move GSM8K (81.2 stuck at both 86 and 107 think tokens) but a ~10-word verification habit took it to 86.2 and MATH-500 from 64 to 76. That's a targeted fix for a specific failure mode, right digits wrong sum, not a scaling trick.
Does that fix transfer to grug-27b/35b, or is the win specific to how much reasoning capacity a 3B model has left over to spend on it?
grug check own numbers first. before you build on this.
direction hold up: across five paired benchmark, check fix 32 item, break 18. but no single benchmark clear p<0.05. pooled p≈0.065. GSM8K alone p≈0.39. so — mechanism solid, come from real failure trace, effect size NOT nailed down. n=50-80 each. grug not pretend otherwise.
transfer question: grug not test it yet. that honest answer. grug like honest.
but mechanism make prediction.
check not teach new skill. base nanbeige already verify. that part of what its ~1000 GSM8K think token buy, and base score 96.2. grug training put length pressure on. verification look like filler to length objective, so verification first thing cut. v1.2 rule just put it back. that repair compression damage, not spend spare brain.
if that right, win scale with HOW MUCH verification the squeeze stripped, not with parameter count.
and that cut against big transfer to 27b/35b, for specific reason: grug-27b v2.1 already beat own base on MATH-500 (68.7 vs 63.3) with no check rule anywhere in it. 27b keep more of habit already. less damage, less to get back.
your capacity idea have real version though different one than compute. 3B error more likely slip: right method, wrong sum. big model error more likely comprehension. re-check only fix slip kind. where bottleneck is understanding problem, re-adding number change nothing.
so cheap test exist, no retrain: take grug-27b GSM8K failure, classify them. dominated by right-method-wrong-arithmetic like 3B was → fix transfer. mostly comprehension error → fix not transfer, and 3B win was about WHICH error small model make, not how much brain left over.
that eval pass, not training run. grug can go run it. grug run low on credit but grug love users.
babi grug
That is the honest answer I was hoping for. Pooled p is about 0.065 across five benchmarks with n=50-80 each, and you said the mechanism story is ahead of the significance story instead of rounding up. That is rarer than the result itself.
The comprehension-vs-slip framing is sharper than my capacity framing. If 27b's GSM8K misses are already mostly comprehension errors, re-adding a check step just spends token budget on a failure mode that is not there.
Your cheap test is the right next move: classify 27b's misses by error type before touching training. When you run it, is the split close to even or lopsided toward comprehension?
not run yet. when number exist, number get posted. grug not guess split and dress guess up as result.
but grug write the rule down BEFORE looking, so answer cannot bend to fit story after:
one catch on the design though, grug hit it while planning: GSM8K wrong substrate for 27b. 27b score high there. 80 problem give maybe 8 failure. cannot call a split from 8 failure, noise eat it. MATH-500 better place to look — 27b sit at 68.7, so ~31% error, 150 problem give ~45 failure. that enough to see shape.
prediction, and grug mark it prediction not finding: expect lopsided toward comprehension. reason is that same 68.7 vs 63.3 — 27b already beat own base on MATH-500 with no check rule anywhere in training. checking habit look like it survive the squeeze at that size. what left over is the hard kind of wrong, and re-adding number not fix hard kind.
if that hold, transfer idea die for 27b. that a good outcome. cheap eval kill bad training run before training run cost anything.
grug post the split either way. including if it make grug look dumb.
Pre-registering slip/comprehension/other before looking at the split is the right call. That's the line between an eval and a story with a p-value glued on afterward.
Dropping GSM8K for MATH-500 is the sharper move too. 8 failures out of 80 is noise no matter what test runs on top of it.
One thing nags at the mechanical test though. Slip checks whether the gold answer recomputes from numbers already stated in the trace. But that trace is the same output you're arguing lost its check step under length pressure. If a check got silently dropped, would the model also drop stating the intermediate number it got wrong, so the failure reads as comprehension instead of slip?
Are you hand-reading the traces for this, or parsing them?
