code/push_release.py

"""
Public release of the v6 weights as **oddadmix/Nawah-Math-Reasoning**.

This is `push_model_v6.py` retargeted at a public repo under the release name. The differences
that matter:

  * `private=False` — this is the release, unlike every rung of the v1→v6 ladder.
  * The card is written for someone who has never seen the ladder. v1–v5 stay private, so the
    comparison table names them but does not link them, and the text does not send readers to
    repos they cannot open.
  * The training code ships **inside the repo** under `code/`, so the recipe and the weights are
    one link. `code/README.md` is generated by `build_release_code.py`.

Every number in the card is still read from an eval file on disk — nothing is typed by hand:

    Nawah-Reasoning-v6/eval_reasoning.json   v6 on the 1,800-row v6 mix eval (by_source, 4 sources)
    Nawah-Reasoning-v6/eval_on_synth.json    v6 on the same 1,000-row synth set v4/v5 used
    Nawah-Reasoning-v5/eval_reasoning.json   v5 on the 1,400-row v5 mix eval
    Nawah-Reasoning-v5/eval_on_synth.json    v5, same 1,000 synth rows
    Nawah-Reasoning-v5/eval_on_rel.json      v5 on v6's 400 held-out relational rows (the baseline)
    Nawah-Reasoning-v4/eval_on_mix.json      v4 on v3's mix eval
    Nawah-Reasoning-v4/eval_on_synth.json    v4, same 1,000 synth rows
    Nawah-Reasoning-v3/eval_reasoning.json   v3 on the same mix eval
    Nawah-Reasoning-v3/eval_on_synth.json    v3, same 1,000 synth rows

Usage: python push_release.py [--repo oddadmix/Nawah-Math-Reasoning] [--dry-run] [--no-code]
"""
import argparse
import json
from pathlib import Path

from huggingface_hub import HfApi

MODEL_DIR = Path("./Nawah-Reasoning-v6")
V5_DIR = Path("./Nawah-Reasoning-v5")
V4_DIR = Path("./Nawah-Reasoning-v4")
V3_DIR = Path("./Nawah-Reasoning-v3")
CODE_DIR = Path("./release_code")
BASE = "oddadmix/50M-2048-Emhotob"
DEMO = "oddadmix/Nawah-Math-Reasoning-Demo"

CARD = """---
license: apache-2.0
language:
- ar
base_model: {base}
datasets:
- oddadmix/arabic-math-reasoning-synth
- oddadmix/gsm8k-reasoning-ar
- Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset
library_name: transformers
pipeline_tag: text-generation
tags:
- arabic
- reasoning
- chain-of-thought
- math
- gsm8k
- small-language-model
- slm
- llama
- sft
---

# Nawah-Math-Reasoning — نموذج استدلال رياضي عربي

A **{params:.1f}M-parameter** Arabic math reasoning model. It writes its derivation step by step
inside `<think>…</think>`, then gives the answer. It is small enough to run on a CPU.

> **بالعربية:** نموذج عربي صغير (~{params:.0f} مليون معامل) لحل المسائل الحسابية: يكتب خطوات
> تفكيره داخل وسم `<think>` ثم يعطي الإجابة. صغير بما يكفي ليعمل على المعالج (CPU).

| | |
|---|---|
| 🤗 **Demo** | [`{demo}`](https://huggingface.co/spaces/{demo}) |
| 🧩 **Base model** | [`{base}`](https://huggingface.co/{base}) — Llama architecture, 12 layers, hidden 512, 2048 ctx, pre-trained from scratch on ~20B Arabic tokens |
| 📚 **Data** | [`arabic-math-reasoning-synth`](https://huggingface.co/datasets/oddadmix/arabic-math-reasoning-synth) · [`gsm8k-reasoning-ar`](https://huggingface.co/datasets/oddadmix/gsm8k-reasoning-ar) · [`Arabic_Reasoning_Dataset`](https://huggingface.co/datasets/Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset) |
| 🛠️ **Training code** | [`code/`](https://huggingface.co/{repo}/tree/main/code) in this repo — data generation, translation, SFT, eval, GRPO |
| 🔤 **Vocab** | {vocab} (4 chat/reasoning tokens added to the 32000 base vocab) |

## Results

Number agreement, greedy decoding. **Every cell is measured on identical held-out rows.** The
`Arabic_Reasoning` and `GSM8K-ar` rows are the eval splits fixed at the start of the project and
never re-drawn; the synthetic rows are pinned to the same 1,000 items every earlier version was
scored on.

The `v3 / v4 / v5` columns are internal development runs, kept here because they are what makes
the release number mean something. They are not published — the numbers are, so the ablation is
readable without them.

| eval set | n | v3 | v4 | v5 | **release** |
|---|---:|---:|---:|---:|---:|
| GSM8K-ar | {g_n} | {v3_gsm:.1f}% | {v4_gsm:.1f}% | {v5_gsm:.1f}% | **{v6_gsm:.1f}%** |
| Arabic_Reasoning | {a_n} | {v3_ar:.1f}% | {v4_ar:.1f}% | **{v5_ar:.1f}%** | {v6_ar:.1f}% |
| synthetic math | 1000 | {v3_synth:.1f}% | {v4_synth:.1f}% | {v5_synth:.1f}% | **{v6_synth:.1f}%** |
| **synthetic relational** | {r_n} | — | — | {v5_rel:.1f}% | **{v6_rel:.1f}%** |

**The relational row is what this release adds.** On problems whose difficulty is the *relation*
between quantities (`ضعف`, `نصف`, `أكثر بـ…`) rather than the arithmetic, it scores
**{v6_rel:.1f}%** where the previous run scores {v5_rel:.1f}% — a **{rel_gain:+.1f} point** gain and
the largest single-cell move anywhere in the development ladder. It did not cost the other
distributions: GSM8K-ar is simultaneously the best of the series at **{v6_gsm:.1f}%**, and
synthetic math gains {synth_delta:+.1f}.

The one regression is `Arabic_Reasoning` at **{ar_delta:+.1f}** against v5 — on {a_n} rows that is
close to sampling noise, but it is the second consecutive mix where this column is the give.

| detail | GSM8K-ar | Arabic_Reasoning | synth math | synth relational |
|---|---:|---:|---:|---:|
| final-answer number correct | {g_primary:.1f}% | {a_primary:.1f}% | {s_primary:.1f}% | {r_primary:.1f}% |
| all numbers match | {v6_gsm:.1f}% | {v6_ar:.1f}% | {s_nums:.1f}% | {v6_rel:.1f}% |
| well-formed `<think>` + answer | {g_well:.1f}% | {a_well:.1f}% | {s_well:.1f}% | {r_well:.1f}% |
| mean reasoning length | {g_tokens:.0f} tok | {a_tokens:.0f} tok | {s_tokens:.0f} tok | {r_tokens:.0f} tok |

*(the synth-math column here is the 400-row mix cell; the {v6_synth:.1f}% in the table above is the
1,000-row set used for the cross-model comparison.)*

Reproduce any cell with `code/eval_reasoning.py` — it is the same script for every model and every
row, which is the only reason these are comparable.

### The final checkpoint ships, and eval loss disagrees

Loss bottoms at **{best_loss:.4f}** (epoch {best_epoch:.2f}) and rises to **{shipped_loss:.4f}** by
epoch {epochs} — yet the epoch-{epochs} weights are the better model. This was measured directly on
an earlier run whose corpus contained **no repeated rows**, which rules out memorisation: the
minimum-loss checkpoint scored 30.9% where the final scored 35.6%. It happened on four consecutive
runs. `train_reasoning.py` therefore takes `LOAD_BEST=0`, and that is not an oversight.

## Training mix

{train_n:,} rows, {tok_per_epoch:.1f}M tokens/epoch:

| source | rows | tokens/epoch | share |
|---|---:|---:|---:|
| [`oddadmix/arabic-math-reasoning-synth`](https://huggingface.co/datasets/oddadmix/arabic-math-reasoning-synth) | {synth_rows:,} | {synth_tok:.2f}M | {synth_share:.1f}% |
| [`oddadmix/gsm8k-reasoning-ar`](https://huggingface.co/datasets/oddadmix/gsm8k-reasoning-ar) | {gsm_rows:,} | {gsm_tok:.2f}M | {gsm_share:.1f}% |
| [`Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset`](https://huggingface.co/datasets/Omartificial-Intelligence-Space/Arabic_Reasoning_Dataset) | {ar_rows:,} (5,536 × 3) | {ar_tok:.2f}M | {ar_share:.1f}% |

Of the synthetic corpus's {corpus_rows:,} rows, {rel_rows:,} are **relational** problems generated
specifically for this release, after a `pass@k` diagnostic showed the previous model went 0/8 on
`ضعف`-style problems and a corpus audit found the relation appears in only 1.34% of rows. The
synthetic eval split was **pinned, not re-drawn** when those rows were added: re-shuffling would
have moved 1,955 of the 2,000 previously held-out items into train, turning that column into a
memorisation score.

Full fine-tune from the base (not from the previous version). Loss on the assistant turn only, user
prompt masked with `-100`. `Arabic_Reasoning` is ~25× smaller than GSM8K, so it is repeated 3×.

| | |
|---|---|
| epochs | {epochs} ({steps:,} steps) |
| effective batch | 64 |
| learning rate | 3e-4 cosine, 200 warmup steps |
| max length | 768 tokens (mix p100 is {p100} — nothing truncated) |
| precision | bf16 |
| checkpoint | final (`load_best_model_at_end` disabled — it picks the worse model) |
| hardware | 1× RTX A6000, ~{minutes} min |

## Usage

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "{repo}"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16).eval()

messages = [{{"role": "user", "content": "اشترى خالد 4 دفاتر بسعر 15 جنيهًا للدفتر، ودفع بورقة 100 جنيه. كم المبلغ المتبقي؟"}}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
ids = tok(prompt, return_tensors="pt")

out = model.generate(**ids, max_new_tokens=384, do_sample=False)
print(tok.decode(out[0][ids["input_ids"].shape[1]:], skip_special_tokens=False))
```

Split the parts with `re.match(r"\\s*<think>(.*?)</think>(.*)", completion, re.S)`.
Decode with `skip_special_tokens=False` — `<think>` and `</think>` are real tokens in this
tokenizer, and stripping them destroys the split.

It is **single-turn**: one user message per call. Chat history is out of distribution.

**Answer style is not something you can request.** The three corpora disagree — GSM8K rows end in a
bare numeral, the other two in an `إذن، …` sentence — and arithmetic word problems look alike in
all of them, so the model picks a style per prompt. **Score it on number agreement, not exact
string match**, and parse the answer by extracting its numbers.

## Limitations

At ~{params:.0f}M parameters this is a **proof of concept**, and the honest headline is the
synthetic columns — **{v6_synth:.1f}%** and **{v6_rel:.1f}%** on multi-step problems, well below the
{v6_gsm:.1f}% it scores on GSM8K's narrower phrasing. Arithmetic is the dominant failure mode: the
reasoning is usually structurally right, one computation step is wrong, and the model then stays
faithful to its own bad number.

Each corpus brings its own defect. The GSM8K half is machine-translated, its 140,969 rows expanding
from only 2,814 question patterns, so that score partly reflects narrow phrasing. The synthetic
half is verified for **arithmetic, not for sense** — rows survive where every equation checks out
but a step introduces an entity never mentioned, or the answer resolves the reverse of what was
asked. The `Arabic_Reasoning` half excludes open-ended expository rows (they have no final answer
to place after `</think>`), so expository prompts remain out of distribution.

Everything is MSA; the synthetic corpus's region axis sets currency and context, not dialect. The
Arabic inherits source artifacts including inconsistent gender agreement. Its reasoning trace is
not a faithful account of any internal computation. Do not use it for anything consequential.

## Citation

```bibtex
@misc{{nawah_math_reasoning_2026,
  title  = {{Nawah-Math-Reasoning: a 52M-parameter Arabic chain-of-thought math model}},
  author = {{Ahmed Wasfy}},
  year   = {{2026}},
  url    = {{https://huggingface.co/{repo}}}
}}
```
"""


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--repo", default="oddadmix/Nawah-Math-Reasoning")
    ap.add_argument("--dry-run", action="store_true")
    ap.add_argument("--no-code", action="store_true", help="skip the code/ upload")
    ap.add_argument("--minutes", default="85")
    args = ap.parse_args()

    def summary(p):
        return json.loads(Path(p).read_text(encoding="utf-8"))["summary"]

    v6 = summary(MODEL_DIR / "eval_reasoning.json")
    v6_synth = summary(MODEL_DIR / "eval_on_synth.json")
    v5_mix = summary(V5_DIR / "eval_reasoning.json")
    v5_synth = summary(V5_DIR / "eval_on_synth.json")
    v5_rel = summary(V5_DIR / "eval_on_rel.json")
    v4_mix, v4_synth = summary(V4_DIR / "eval_on_mix.json"), summary(V4_DIR / "eval_on_synth.json")
    v3_mix, v3_synth = summary(V3_DIR / "eval_reasoning.json"), summary(V3_DIR / "eval_on_synth.json")

    g, a, s, r = (v6["by_source"]["gsm8k_ar"], v6["by_source"]["arabic_reasoning"],
                  v6["by_source"]["synth_math_ar"], v6["by_source"]["synth_relational_ar"])

    tm = json.loads(Path(MODEL_DIR, "train_metrics.json").read_text(encoding="utf-8"))
    cfg = json.loads(Path(MODEL_DIR, "config.json").read_text(encoding="utf-8"))
    evals = [h for h in tm["log_history"] if "eval_loss" in h]
    last = max(h["step"] for h in evals)
    shipped_loss = next(h["eval_loss"] for h in evals if h["step"] == last)
    best = min(evals, key=lambda h: h["eval_loss"])

    v3_gsm, v4_gsm, v5_gsm = (v3_mix["by_source"]["gsm8k_ar"], v4_mix["by_source"]["gsm8k_ar"],
                              v5_mix["by_source"]["gsm8k_ar"])
    v3_ar, v4_ar, v5_ar = (v3_mix["by_source"]["arabic_reasoning"],
                           v4_mix["by_source"]["arabic_reasoning"],
                           v5_mix["by_source"]["arabic_reasoning"])

    # measured from data_v6_sft/train.jsonl with the base tokenizer (+12 tok/row of chat scaffold)
    MIX = {"synth_math_ar": (118062, 16.79, 53.9), "gsm8k_ar": (140969, 11.88, 38.2),
           "arabic_reasoning": (16608, 2.45, 7.9)}

    card = CARD.format(
        repo=args.repo, base=BASE, demo=DEMO, params=51.79, vocab=cfg["vocab_size"],
        g_n=g["n"], a_n=a["n"], r_n=r["n"],
        v3_gsm=v3_gsm["numbers_match_pct"], v4_gsm=v4_gsm["numbers_match_pct"],
        v5_gsm=v5_gsm["numbers_match_pct"], v6_gsm=g["numbers_match_pct"],
        v3_ar=v3_ar["numbers_match_pct"], v4_ar=v4_ar["numbers_match_pct"],
        v5_ar=v5_ar["numbers_match_pct"], v6_ar=a["numbers_match_pct"],
        v3_synth=v3_synth["numbers_match_pct"], v4_synth=v4_synth["numbers_match_pct"],
        v5_synth=v5_synth["numbers_match_pct"], v6_synth=v6_synth["numbers_match_pct"],
        v5_rel=v5_rel["numbers_match_pct"], v6_rel=r["numbers_match_pct"],
        rel_gain=r["numbers_match_pct"] - v5_rel["numbers_match_pct"],
        ar_delta=a["numbers_match_pct"] - v5_ar["numbers_match_pct"],
        synth_delta=v6_synth["numbers_match_pct"] - v5_synth["numbers_match_pct"],
        g_primary=g["primary_number_match_pct"], a_primary=a["primary_number_match_pct"],
        s_primary=s["primary_number_match_pct"], r_primary=r["primary_number_match_pct"],
        s_nums=s["numbers_match_pct"],
        g_well=g["well_formed_pct"], a_well=a["well_formed_pct"],
        s_well=s["well_formed_pct"], r_well=r["well_formed_pct"],
        g_tokens=g["mean_reasoning_tokens"], a_tokens=a["mean_reasoning_tokens"],
        s_tokens=s["mean_reasoning_tokens"], r_tokens=r["mean_reasoning_tokens"],
        best_loss=best["eval_loss"], best_epoch=best["epoch"], shipped_loss=shipped_loss,
        epochs=round(max(h.get("epoch", 0) for h in tm["log_history"])),
        steps=max(h.get("step", 0) for h in tm["log_history"]),
        train_n=sum(v[0] for v in MIX.values()),
        tok_per_epoch=sum(v[1] for v in MIX.values()),
        synth_rows=MIX["synth_math_ar"][0], synth_tok=MIX["synth_math_ar"][1],
        synth_share=MIX["synth_math_ar"][2],
        gsm_rows=MIX["gsm8k_ar"][0], gsm_tok=MIX["gsm8k_ar"][1], gsm_share=MIX["gsm8k_ar"][2],
        ar_rows=MIX["arabic_reasoning"][0], ar_tok=MIX["arabic_reasoning"][1],
        ar_share=MIX["arabic_reasoning"][2],
        corpus_rows=120462, rel_rows=20139, p100=703, minutes=args.minutes,
    )
    out = Path("release_README.md")
    out.write_text(card, encoding="utf-8")
    print(f"[+] wrote {out} ({len(card)} chars)")

    if args.dry_run:
        print("[dry-run] not pushing")
        return

    api = HfApi()
    api.create_repo(args.repo, private=False, exist_ok=True)

    # Weights first, then the card, so the repo is never public-and-uncarded for long.
    api.upload_folder(
        folder_path=str(MODEL_DIR), repo_id=args.repo,
        # eval_on_synth.json ships: it is the 1,000-row cross-model synth cell the card quotes.
        # eval_on_v6mix.json does not — it is byte-identical to eval_reasoning.json.
        ignore_patterns=["checkpoint-*/*", "checkpoint-*", "eval_on_v6mix.json", "README.md"],
        commit_message="Nawah-Math-Reasoning: weights, tokenizer, eval + training metrics")
    api.upload_file(path_or_fileobj=str(out), path_in_repo="README.md", repo_id=args.repo,
                    commit_message="model card")

    if not args.no_code:
        if not CODE_DIR.is_dir():
            raise SystemExit(f"{CODE_DIR} missing — run build_release_code.py first")
        api.upload_folder(folder_path=str(CODE_DIR), repo_id=args.repo, path_in_repo="code",
                          commit_message="training code: data generation, SFT, eval, GRPO")

    print(f"[+] https://huggingface.co/{args.repo}")


if __name__ == "__main__":
    main()