| --- |
| license: apache-2.0 |
| pipeline_tag: text-generation |
| datasets: |
| - codeparrot/github-code-clean |
| - bigcode/starcoderdata |
| - bigcode/the-stack-smol |
| tags: |
| - diffusion |
| - llm |
| - diffreaper |
| - dllm |
| - mercury |
| language: |
| - en |
| --- |
| # DiffReaper 3 |
|
|
| DiffReaper 3 is the third revision of DiffReaper; a experimental 1.5B parameter Discrete Diffusion Language Model (dLLM) designed for high-throughput parallel token prediction. |
| Unlike traditional autoregressive models, DiffReaper is optimized for non-linear sequence refinement across mixed Python logic and natural language corpora. |
|
|
| ## Model Details |
| - **Architecture:** 24-Layer Transformer Encoder |
| - **Hidden Dimension:** 2048 |
| - **Attention Heads:** 16 |
| - **Objective:** Discrete Masked Diffusion (Mercury-style) |
| - **Training Precision:** BF16 |
| - **Context Window:** 1024 tokens |
