Logicvex AI

@simonko912 That 0.41B scale is actually a sweet spot for practical edge models. Getting basic coding, math, and instruction following at 410M with a clean Llama arch is solid work. Do you have a link to the model card or repo? Would love to test it out.

@simonko912 Honestly, Chinchilla-optimal or undertrained beats over-baked weights any day. At least your parameters can still learn new things. What models or architectures are you working on lately? Always cool connecting with others experimenting in this space.

It overtrains, not overfits.

Overtraining is the mechanism, parameter rigidity and quant collapse are the results. Playing semantic word games doesn't change loss landscape physics.

Admitting you're overtraining 2.9M params on 75B tokens just concedes the entire point.

@Banaxi-Tech Shifting goalposts from "pipeline validation probe" to "it will beat Photon on benchmark scores" says it all.

Comparing a clean, native 1M LLaMA model built with standard architecture and zero custom code (trust_remote_code=False) to a 2.9M custom gimmick overtrained on 75B tokens to game multiple-choice evals is apples to oranges.

Also, claiming 75B tokens on 2.9M params "can't overfit" shows a complete misunderstanding of loss landscape physics. Driving 75B tokens through 2.9M params over-bakes the weights into brittle minima, destroying param plasticity and weight stability.

Good luck gaming the leaderboard. We'll stick to building standard, deployable architectures.

Taking a 26,000:1 token-to-parameter ratio on a 2.9M micro-model and linearly extrapolating it to a 70B scale to invent an absurd "1.8 quadrillion token" requirement is a fundamentally dishonest argument that intentionally confuses compute-optimal deployment with capacity-optimal research probing. Nobody is proposing a quadrillion-token run for a frontier model. When we talk about "maximum intelligence per parameter," that is not a goalpost shift, as it is the literal definition of capacity-optimal research. Pushing a tiny, fixed parameter footprint to see how much structural signal it can absorb before hitting diminishing returns. Using a fast, cheap 2.9M model to test the asymptotic limits of parameter density and using it to validate their training pipeline are two sides of the exact same coin.

Dismissing extreme token density on a tiny model as mere "bottlenecked memorization" reveals your misunderstanding of how modern data curation and entropy work. If you feed 75B tokens of unrefined, low-quality web noise into a 2.9M model, its loss curve flatlines almost immediately. However, when you train on heavily filtered, high-entropy, synthetic data (which is actually the exact methodology validated by work like Microsoft's Phi series), a micro-architecture continuously extracts logical primitives, syntactic structures, and core reasoning abstractions over tens of billions of tokens. Pushing a micro-model to these extreme ratios was never an exercise in memorization, it is an aggressive acid test for data quality. It proves whether filtering algorithms yield actual informational signal or just noise, providing transferable insights into how novel optimizers like Muon and aggressive learning rates perform under extreme compression.

Finally, judging a 2.9M-parameter research probe by its SFT plasticity, 4-bit quantization behavior, or tensor parallelism compatibility is like evaluating an experimental go-kart by the payload standards of a commercial semi-truck. These are production deployment metrics, not research criteria. Complaining about trust_remote_code=True or an uncompiled custom gate on a toy model that runs on a single consumer GPU in a few hours is pure bikeshedding. Good ML research prioritizes rapid iteration in native PyTorch to validate mathematical convergence and architectural stability, like pairing Muon at a 2.2e-2 learning rate with an XSA refresh gate, before wasting engineering cycles writing custom Triton kernels or optimizing multi-node parallelism for billion-parameter runs. This micro-model does exactly what it was designed to do: cheaply prove out architectural and data innovations at the absolute limit so we can scale them up with confidence.

Citing Microsoft Phi to defend a 25.8k:1 token to param ratio makes zero sense when you look at the actual numbers.

Microsoft's highest ratio in the entire family is Phi-3.5-mini: 3.8B params on 3.4T tokens. That's an 890:1 ratio. The 2.9M model you're defending runs at 25,862:1. That's 29x more saturated than Phi-3.5, and 224x more saturated than Phi-1.5. You aren't citing an established methodology, you're just name-dropping Phi to cover up data drowning.

Claiming a 2.9M model "continuously extracts logical primitives and core reasoning abstractions" over 75B tokens is also pure pseudoscience. At 2.9M params, a model's hidden dimension is usually under 200, placing it deep inside the Softmax Bottleneck. The rank of its hidden space is physically too narrow to represent high-rank logical abstractions. Its capacity flatlines in the first couple billion tokens. The remaining 70+ billion tokens aren't extracting "reasoning", they're just grinding saturated weights against surface text statistics.

Your core argument that this micro-model works as a "research probe" to validate data and architectures for 70B scale is dead wrong.

The literature on micro-scale models proves sub-20M networks don't scale predictively to LLMs. "Scaling Laws in the Tiny Regime" (arXiv 2026) shows standard power laws break down below 20M params, local exponents decay, and error structures change completely. Tiny models just memorize easy patterns. As Wang et al. (2025) showed, micro-models test surface memorization, while larger models execute implicit reasoning. Overfitting 75B tokens onto 2.9M params tells you zero about how that data or architecture behaves at frontier scale.

Calling quantization fragility a "production deployment metric" misses the entire point. Kumar et al. (2024) in "Scaling Laws for Precision" proved that a 25k+ D/N ratio guarantees power-law degradation under low-bit compression. When weights are so brittle that dropping precision destroys them, that isn't a deployment detail, it's mathematical proof of extreme over-fitting.

They didn't build a capacity-optimal research probe. You're defending an over-compressed benchmark gimmick that can't be quantized, can't be fine-tuned, forces trust_remote_code=True, and proves nothing about larger models.

I dont know what the purpose of this comment is, the goal of these models were never "here is a general-purpose, 4-bit quantizable foundation model to replace GPT in your local SFT stack", framing it like that is a complete stretch.

The goal of small models like these are to answer a question like "How much signal can a tiny model retain when trained on curated data and modern architectural efficiency?"

Critiquing a 2.9M-parameter model for lacking SFT plasticity or parameter redundancy completely ignores the point. Pushing parameter efficiency to its theoretical limits isn't meant to deliver a production-ready edge model, it is for validating their data pipelines, filtering algorithms, and training loops.

Nobody is telling that you that you should deploy a 2.9M model, it's that better data and smarter architecture yield vastly superior intelligence per parameter. Those exact data and training breakthroughs are what scale up directly to billions or trillions of parameters.

Moving the goalposts from "maximum intelligence per parameter" to "it's just a pipeline validation probe" the moment the math gets checked is wild.

Claiming a 26,000 tokens/param ratio "scales directly to billions or trillions" shows a fundamental misunderstanding of scaling laws. At a 26k:1 ratio, a 70B model would require 1.8 quadrillion tokens.

Extreme overtraining at 2.9M params measures bottlenecked memorization under artificial compression, not generalizable data signal. Overfitting a micro-model on 75B tokens tells you zero about how that data or architecture behaves at frontier scale.

And if the goal is "pipeline validation", slapping an uncompiled custom gate on a toy model that breaks tensor parallelism and forces trust_remote_code=True validates nothing except benchmark gaming.

75B tokens on 2.9M params is ~25.8k tokens/param. That's not "max intelligence per param", it's a textbook saturation trap.

Drowning 2.9M weights in 75B tokens destroys SFT plasticity, causes instant 4-bit quant collapse, and leaves zero param redundancy. Add an uncompiled custom gate with trust_remote_code=True and you've paid every single Gimmick Tax.

You're announcing a static lookup table that can't be fine-tuned, quantized, or safely deployed. Great bench scores on paper, zero real-world utility.