Free Access to Mythos Translation AI

This repository is publicly accessible, but you have to accept the conditions to access its files and content.

Mythos-2.04B-MoE is 100% free to use with zero billing or fees! To help grow this open project and support further development, please click the ❤️ Like button at the top of this repository.

Mythos-2.04B-MoE

This repository contains model weights and configuration files for the pre-trained model in the PyTorch and Hugging Face format.

These artifacts are compatible with PyTorch, Hugging Face Transformers, vLLM, SGLang, TokenSpeed, etc.

Mythos-2.04B-MoE is 100% free for everyone! If you find this model helpful, please support our open-source research by clicking the ❤️ Like button at the top right of this page!

Following the development of modern multilingual architectures, we are pleased to introduce Mythos-2.04B-MoE, the most capable generation in the Mythos open-model family to date.

Built on the architectural foundation of Sparse Mixture-of-Experts (SMoE), Mythos-2.04B-MoE delivers substantial gains across translation quality, low-resource language understanding, zero-shot cross-lingual transfer, and long-horizon document translation. Mythos-2.04B-MoE brings these advances to a compact, deployment-friendly sparse model: a native multilingual translation model that understands 500+ global languages, designed to carry complex sentence, paragraph, and multi-page document translations through to completion with greater reliability.

Comparison with Open and Proprietary Translation Models

Following the architectural comparison standards established by frontier open models, the table below compares the structural specifications, context capacity, language coverage, and hardware footprint of Mythos-2.04B-MoE against prevailing state-of-the-art models:

Specification	Mythos‑2.04B‑MoE	TranslateGemma‑27B	TranslateGemma‑7B	NLLB‑200 (3.3B)	Google Cloud API
Architecture	Sparse MoE (8E, Top‑2)	Dense Transformer	Dense Transformer	Dense Enc-Dec	Proprietary LLM
# Total Params	2.04B	27.2B	7.0B	3.3B	Closed
# Active Params	678.7M	27.2B	7.0B	3.3B	Closed
Context Window	8,192 tokens	2,048 tokens	2,048 tokens	1,024 tokens	Varies
Supported Languages	500+	55	55	200	189
Min Inference VRAM	4 GB	56 GB	16 GB	8 GB	Managed Cloud

Key Architectural Insights:

Unrivaled Multilingual Breadth (500+ Languages vs. 55 - 243):
- While Google Cloud supports 189 languages, Google Translate supports 243, and Google TranslateGemma-27B benchmarks 55 languages, Mythos supports 500+ languages natively.
- Over 250+ underserved African, Oceanic, Indigenous American, and Regional South/Central Asian dialects that have zero coverage in mainstream models are fully supported by Mythos with zero-shot cross-lingual transfer.
Long-Horizon Context Handling (8,192 Tokens vs. 1k - 2k):
- Specialized translation models (NLLB-200 at 1,024 tokens, TranslateGemma at 2,048 tokens) suffer from context fragmentation when translating multi-paragraph documents, technical manuals, or legal contracts.
- Mythos features a native 8,192-token context window with Block-Diagonal Attention Packing, preserving document discourse, technical terminology, and character pronoun consistency across entire chapters.
Radical Compute and Memory Efficiency (Sparse MoE):
- Running TranslateGemma-27B requires 27 Billion active parameters per token, demanding expensive multi-GPU enterprise infrastructure.
- Mythos utilizes Sparse Mixture-of-Experts with Top-2 routing across 8 SwiGLU experts. It activates only 678.7 Million parameters per token, achieving the capacity of a multi-billion parameter model while running at high token throughput on modest hardware ($10/month cloud VPS, consumer RTX GPUs, or CPU).

Mythos-2.04B-MoE Highlights

Mythos-2.04B-MoE features the following enhancements:

Core Translation Capabilities: Comprehensive improvements across cross-lingual semantic alignment, high-resource language fluency, and zero-shot translation paths across low-resource dialects.
Sparse MoE Efficiency: 8 specialized SwiGLU expert feedforwards per block with Top-2 dynamic routing, delivering 2.04B total parameter knowledge capacity while requiring only 678M active parameters per token.
Long-Horizon Context Handling: Dedicated 8,192-token context window with Block-Diagonal Document Packing, effortlessly handling single sentences, multi-sentence paragraphs, and large documents without truncation.
Vocabulary & Script Coverage: 128,000-token Byte-Level BPE subword vocabulary natively supporting all major writing systems (Latin, Cyrillic, Arabic, Devanagari, Hanzi, Kana, Hangul, African, and Indigenous scripts).
Downstream Compatibility: Standard Encoder-Decoder interface compatible with popular inference harnesses and distributed frameworks.

Model Overview

Type: Encoder-Decoder Multilingual Translation Model with Sparse Mixture-of-Experts (SMoE)
Training Stage: Pre-training
Language Model Architecture:
- Number of Parameters: 2.04B (2,037,643,264 Total Parameters)
- Active Parameters / Token: 678.7M (678,688,768 Active Parameters)
- Hidden Dimension: 1,024
- Token Embedding: 128,000 (Byte-Level BPE)
- Number of Layers: 24 Layers (12 Encoder Layers + 12 Decoder Layers)
- Hidden Layout: 24 × (GQA Attention $\to$ SwiGLU MoE FFN)
- Gated Attention (GQA):
  - Number of Attention Heads: 16 for Q and 4 for KV (4× KV-Cache Compression)
  - Head Dimension: 64
  - Rotary Position Embedding (RoPE) Dimension: 64 ($\theta = 100,000.0$)
- Feed-Forward Network (Sparse MoE):
  - Number of Experts: 8 Experts per block
  - Routing Mechanism: Top-2 Active Experts with Load Balancing + Router Z-Loss ($1\times 10^{-4}$)
  - Intermediate Dimension (per Expert): 3,072 (SwiGLU)
- LM Output: 128,000 (Triton Liger-Fused Cross-Entropy)
- Context Length: 8,192 tokens natively with Block-Diagonal Document Packing

Quickstart & Inference

1. Installation

pip install torch transformers tokenizers sacrebleu unbabel-comet

2. Python Inference Code

import torch
from tokenizers import Tokenizer

# Load Multilingual Tokenizer
tokenizer = Tokenizer.from_file("multilingual_tokenizer.json")
sos_id = tokenizer.token_to_id("[SOS]")
eos_id = tokenizer.token_to_id("[EOS]")
pad_id = tokenizer.token_to_id("[PAD]")

def translate_sentence(model, text, src_lang="eng", tgt_lang="fra", max_len=128, device="cuda:0"):
    clean_tgt = tgt_lang.split("_")[0]
    enc_input = f"<2{clean_tgt}> {text}"
    tokens = tokenizer.encode(enc_input).ids
    src_tensor = torch.tensor([tokens], dtype=torch.long, device=device)
    src_mask = (src_tensor != pad_id).unsqueeze(1).unsqueeze(2)

    with torch.no_grad():
        enc_out = model.encode(src_tensor, src_mask)
        gen_input = torch.tensor([[sos_id, tokenizer.token_to_id(f"<2{clean_tgt}>")]], device=device)
        
        for _ in range(max_len):
            cur_len = gen_input.size(1)
            c_mask = torch.tril(torch.ones((cur_len, cur_len), dtype=torch.bool, device=device)).unsqueeze(0).unsqueeze(0)
            dec_out = model.decode(gen_input, enc_out, src_mask, c_mask)
            logits = model.project(dec_out[:, -1:])
            next_tok = logits.argmax(dim=-1).item()
            if next_tok == eos_id:
                break
            gen_input = torch.cat([gen_input, torch.tensor([[next_tok]], device=device)], dim=1)

    translated_text = tokenizer.decode(gen_input[0].tolist()[2:])
    return translated_text

Training Infrastructure & Engineering

Distributed Engine: PyTorch Fully Sharded Data Parallel (FSDP) in Full-Shard mode (FULL_SHARD).
Mixed Precision: Native 32-bit FP32 Master Weights with accelerated FP16 compute and unscaled logits.
Zero-Host RAM Footprint: Streaming disk-spooler architecture keeping host CPU memory strictly below < 1.0 GB throughout training.
Isolated Subprocess Evaluation: Fully decoupled child-process neural COMET metric evaluation with 0 MB memory leakage into the main training rank.

Evaluation & Benchmarks

(Benchmark results on the 500-Language Universal Validation Suite, COMET-22, BLEU, and ChrF++ will be published here upon completion of the pre-training run.)

Benchmark	Metric	Mythos-2.04B-MoE	Baseline
Fixed 500-Language Suite	Token Accuracy	In Progress	—
COMET-22	Neural Semantic Score	In Progress	—
SacreBLEU	Corpus BLEU	In Progress	—
ChrF++	Character F-score	In Progress	—

Community & Support

Mythos is an open community project dedicated to universal multilingual communication.

Support the Project: Please click the ❤️ Like button at the top of this repository!
Community Discussions: Feel free to open a thread on the Community tab for feature requests, language additions, or questions.
Organization: Adithyan AI
Repository: AdithyanAI/Mythos_translation

Citation

If you use Mythos-2.04B-MoE or its architecture in your research or applications, please cite:

@misc{mythos2026multilingual,
  author       = {Adithyan AI},
  title        = {Mythos-2.04B-MoE: A Sparse Mixture-of-Experts Large Language Model for Universal Multilingual Translation},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/AdithyanAI/Mythos_translation}}
}

Downloads last month: -; Downloads are not tracked for this model. How to track