Article
NaFlex in timm
rwightman
• • 3
🤝 Open to Collab
Nima Nooshiri
nimanzik
AI & ML interests
None yet
Recent Activity
upvoted an article about 2 months ago
NaFlex in timm updated a collection about 2 months ago
Hugging Face In-Depth Articles updated a collection about 2 months ago
Hugging Face In-Depth ArticlesOrganizations
upvoted an article about 2 months ago
updated 2
collections about 2 months ago
liked a
Space about 2 months ago
upvoted 2
articles 2 months ago
Article
Profiling in PyTorch (Part 1): A Beginner's Guide to torch.profiler
- +3
ariG23498, sayakpaul, sergiopaniego, ror, pcuenq
• • 156 Article
Harness, Scaffold, and the AI Agent Terms Worth Getting Right
sergiopaniego, ariG23498
• • 136 liked a
Space 2 months ago
upvoted an article 3 months ago
Article
KV Caching Explained: Optimizing Transformer Inference Efficiency
not-lain
• • 389 updated a
collection 3 months ago
published a
model 3 months ago
upvoted 4
articles 3 months ago
Article
KV Cache from scratch in nanoVLM
- +3
ariG23498, kashif, lusxvr, andito, pcuenq
• • 120 Article
GGML and llama.cpp join HF to ensure the long-term progress of Local AI
- +4
ggerganov, ngxson, allozaur, lysandre, victor, julien-c
• • 509 Article
Introducing Storage Buckets on the Hugging Face Hub
- +10
Wauplin, coyotte508, XciD, victor, julien-c, lhoestq, pierric, Sylvestre, hlarcher, rajatarya, seanses, assafvayner
• • 197 Article
Running AI agents to automate outreach at scale
nielsr
• • 15 upvoted 2
articles 4 months ago
Article
DeepSeek-V4: a million-token context that agents can actually use
burtenshaw
• • 53 Article
LateOn-Code & ColGrep: LightOn unveils state-of-the-art code retrieval models and code search tooling
lightonai
• • 58 reacted to Kseniase's post with 👍 4 months ago
Post
8419
13+ types of attention mechanisms
Attention mechanisms allow models to dynamically focus on specific parts of their input when performing tasks. In our recent article, we discussed Multi-Head Latent Attention (MLA) in detail and now it's time to summarize other existing types of attention.
Here is a list of 13+ types of attention mechanisms used in AI models:
- Self-attention
- Cross-attention
- Causal attention
- Softmax attention
- Linear attention
- Sliding Window (Local Attention)
- Global attention
- FlashAttention
- Multi-Head Attention (MHA)
- Multi-Query Attention (MQA)
- Grouped-Query Attention (GQA)
- Multi-Head Latent Attention (MLA)
- Interleaved Head Attention (IHA)
Read our post about these core types of attention to learn their specific features and how they work -> 13+ Attention Mechanisms in Transformers and LLMs Explained https://www.turingpost.com/p/attention-types
Attention mechanisms allow models to dynamically focus on specific parts of their input when performing tasks. In our recent article, we discussed Multi-Head Latent Attention (MLA) in detail and now it's time to summarize other existing types of attention.
Here is a list of 13+ types of attention mechanisms used in AI models:
- Self-attention
- Cross-attention
- Causal attention
- Softmax attention
- Linear attention
- Sliding Window (Local Attention)
- Global attention
- FlashAttention
- Multi-Head Attention (MHA)
- Multi-Query Attention (MQA)
- Grouped-Query Attention (GQA)
- Multi-Head Latent Attention (MLA)
- Interleaved Head Attention (IHA)
Read our post about these core types of attention to learn their specific features and how they work -> 13+ Attention Mechanisms in Transformers and LLMs Explained https://www.turingpost.com/p/attention-types
- 1 reply
upvoted an article 4 months ago
Article
DenseOn with the LateOn: Open State-of-the-Art Single and Multi-Vector Models
lightonai
• • 44