Goooooood question...
You’re right to ask and In the current version the comparisons are not paired. Each generation’s three solo tests draw a fresh first-box placement (unseeded), and the “perfect robot” yardstick (4 runs) is also recomputed every generation.
So a 15-point drop mixes a real regression with evaluation noise, and the “three consecutive passes” rule (average of the last 3 generations ≥ 70%) has the same weakness.
The final evaluation of a learned warehouse is different: it runs 20 episodes in which every robot gets the same seeded box order, so that comparison is paired.
Your suggestion makes sense. I’ll fix right not the solo test placements per warehouse (same seeds for every generation and for the yardstick), so the rollback and pass decisions compare like with like. I’ll post the before/after once it’s rerun. Thanks for reading this closely.
U can watch now live how changes afect at https://wh.cawnai.com