Hugging Face's logo
appvoid's picture
🤏 quecto mode

appvoid

appvoid
45 4 173
tanngnle's profile picture lemonzest79's profile picture chhuoymakara's profile picture
·
https://ko-fi.com/appvoid

AI & ML interests

singularity through byte-level tokens and small language models

Recent Activity

repliedto Banaxi-Tech's post about 4 hours ago
Today we wanted to release BananaMind 2 Pico, our smallest model yet at ~0.9M parameters. Instead, we accidentally ran a very expensive experiment on what happens when you push a tiny model way past its useful token budget. Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through. The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting. Follow us for more: https://huggingface.co/BananaMind @vovaRL @Banaxi-Tech Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXP
reacted to Banaxi-Tech's post with 🔥 about 4 hours ago
Today we wanted to release BananaMind 2 Pico, our smallest model yet at ~0.9M parameters. Instead, we accidentally ran a very expensive experiment on what happens when you push a tiny model way past its useful token budget. Short version: we trained on 200B tokens (~222K:1 tokens-per-parameter). The model peaked at 20B tokens with an INT Index of 4.55, then degraded monotonically over the next 160B to 3.31 — a 27% regression. Three of four Open SLM benchmarks were worse at the end of training than they were at 10% through. The useful compute-optimal range for Pico-tier models looks like ~22K–30K tokens per parameter. Ratios like 7K:1, 15K:1, and 22K:1 all work fine — TinyStories and most sub-3M community models sit in this range. Push much further and benchmarks start rotting. Follow us for more: https://huggingface.co/BananaMind @vovaRL @Banaxi-Tech Full writeup with all checkpoints, the Chinchilla-ratio control run, and the schedule-vs-overtraining analysis: https://huggingface.co/blog/Banaxi-Tech/ovdadadadd And if anyone, i dont know the reason why you would, wants the 20B token checkpoint reply and ill upload it as BananaMind 2.1 Pico EXP
posted an update about 23 hours ago
Random corporate secret of tonight: Try overfitting a tiny model on billions of high-quality datapoints: you can't. You can do 100 epochs and see the model still improving. You're welcome.
View all activity

Organizations

ZeroGPU Explorers's profile picture Social Post Explorers's profile picture BananaMind Model Previewers's profile picture Community Early Access Models from appvoid's profile picture