view reply Nice work on the tokenizer optimization! Have you compared it with https://github.com/marcelroed/gigatoken ?
knowledgator/gliclass-modern-large-v3.0 Text Classification • 0.4B • Updated Aug 12, 2025 • 1.26k • 14
opensearch-project/opensearch-neural-sparse-encoding-doc-v3-gte Feature Extraction • 0.1B • Updated Jul 22, 2025 • 9.5k • 18