AtAndDev/example_dataset
Updated • 39
alkinun
AtAndDev
AI & ML interests
decentralize
Recent Activity
liked a dataset about 7 hours ago
common-pile/project_gutenberg_filtered liked a dataset about 17 hours ago
IFM/Pretrain-Behaviors reacted to juiceb0xc0de's post with 🔥 about 17 hours ago
Dude, Where's My Update? I'll tell you where! ~97.6% of my BF16 parameter coordinates didn't move at all, and the ones that did overshot by ~1.33x.
It's nice to do research that doesn't end in disproving yourself once again and moving on to the next subject once in awhile.
Back to the topic, if you've ever wondered why most of your weights are basically ghosting you nearly every step when you store your weights at bf16, Dude, I Measured It.
https://huggingface.co/blog/juiceb0xc0de/intended-and-realized-updates-in-bf16-fine-tuning#dude-wheres-my-updateOrganizations
