Hugging Face's logo

Join the conversation

Join the community of Machine Learners and AI enthusiasts.

Sign Up
Back to feed
Bc-AI 
posted an update 5 days ago
Post
1996
Hello everyone! Happy to say that MiniCoder-1 is now in the instruction tuning phase. It is a 216M~ parameter model trained on 16B tokens. It ran on an RTX 6000 Pro Blackwell gpu for around 24~ hours. Now we will do SFT and launch as beta while we work on the final important DPO and RLHF phases. Our goal is a small extreemly fast on device coding assistant with CoT reasoning* baked in! - Bc-AI on behalf of the Smilyai-Labs team

*it is a small model so the reasoning quality wont be as good obviously!
Anduin1357
•
edited 3 days ago

How are things? Would you maybe explore shipping an optional n-gram embedding layer with the model to help it along, as debuted by Qwen3.8-Flash-Next?

I've heard of the ngram-embedding layer as best being thought of as tab completions which could hint models into better next tokens without going through too much active weights. Basically like a reverse MTP, but the embedding layer is providing knowledge rather than speeding up the decode.

·
Bc-AI

@Anduin1357 ooh i see! i will look into it! We are constantly experimenting with novel architecutres, and if it works we will release minicoder-2 with this feature! thank you for the suggestion!
In this post