Notio2
Note: Refined after the results and learning of my work regarding https://huggingface.co/basically-experimental/Notio-3.7M-RNN-v1 + @Hoglet-33's work regarding their "Pebble" model β https://huggingface.co/collections/basically-ai/pebble - ALL WORK SUBJECT TO CHANGE AS DEVELOPMENT CONTINUES. Discrepancies may remain and unaddressed till completion.
STATUS: INPROGRESS
A deliberately minimal recurrent language model trained on classic English children's books. The model sees only token ids β a u16 stream β and all human-readable rendering happens at runtime through the decoder (ids2txt.py): glyphs map to whitespace effects, boundary marks are dropped from display.
Vocabulary
Vocabulary: 1,024 tokens β 7 single-glyph boundary/whitespace marks (Δ bos, Δ eos, Δ space, Δ newline, plus tab/pad/unk), 94 printable ASCII characters, and 923 subword pieces mined from the 10,000 most common English words (prefix/suffix frequency-scoring, refined by usage until every slot holds its weight).
Content notice
We made our best effort to identify and remove all problematic content β slurs, racial caricature, explicit material, and self-harm mentions β using automated multi-pass scans, boundary verification, and manual sampling. However, no automated or sampled review is perfect, and a small amount of problematic content may remain. The corpus is drawn from public-domain children's literature of 1850β1925 and contains period-typical language, social attitudes, and framing that may not reflect modern standards. Removed materials are preserved in quarantine for review, not deleted. If you find content that concerns you, please report it so it can be removed.
The corpus
~5,000 public-domain children's books from Gutenberg, cleaned down to 2,350 and encoded into one pure id stream.
- Source: the entire Children & Young Adult shelf β Alice, Oz, Potter, Grimm, and thousands more. Free, legal, real literature.
- Cleaning: boilerplate stripped (zero failures), formatting normalized, non-English removed by three independent detectors, duplicates dropped.
- Moral filter: one rule β would you hand this to a child? Slurs, caricature, and explicit content quarantined and verified gone; the era's mild quirks counted and documented.
- Final form: each book wrapped Δβ¦Δ, whitespace as explicit glyphs, encoded to 189,346,083 train ids / 9,997,820 val ids β byte-exact round-trip proven.
TLDR; five thousand public-domain children's books, scrubbed to the standard of "would you hand this to a child to read", merged into one glyph-tagged stream, and encoded to 189 million ids of pure binary.
Architecture
| component | detail |
|---|---|
| embedding | wte + wpe (1024Γ256 each), weights tied to the head |
| core | GRU blocks, d_model=256, pre-LayerNorm + residual (pattern "gg" baseline; "aggg" hybrid adds a sliding-window attention block) |
| head | LayerNorm + lm_head, tied to wte |
| params | 1,315,328 (baseline gg, T=1024) |
| context | block_size 1024, stateful truncated BPTT (k1 = k2 = 1024) |
The stack is modular: layer 0 (data) β layer 1 (embedding) β pattern of blocks β layer last (head), assembled in model/assemble.py.
Training
| hyperparameter | value |
|---|---|
| optimizer | AdamW, grad-clip 1.0 |
| schedule | lr 3e-4, warmup 500 steps, cosine β lr/10 |
| batch | B=32, T=1024 (32,768 tokens/step) |
| BPTT | state carried & detached at window boundaries |
| val | mean of 20 contiguous blocks with carried state |
| sampling | temperature 0.7, top-k 40 |
| early stop | patience on val, best checkpoint kept separately |
Verification
Every artifact is re-checkable read-only:
- scripts/verify.sh β hashes vs known-good manifest, vocab.txt == vocab.bin, split boundary Δ|Δ, id range, training-loop invariants (truncation / statefulness / tape continuity)
- scripts/data_verify.py β full re-tokenization of the corpus must reproduce train+val ids byte-exactly; ids2txt round-trips must reproduce train.bin/val.bin byte-exactly; structure and token census
- Results: 2,350 books balanced Δ/Δ, 1,013/1,024 ids fired, 0 unknown-token usage, 0 raw newline bytes
Usage
scripts/train.py # train baseline (chkpt/final.pt + chkpt/{x}k.pt)
scripts/train.py --check # prove truncation/statefulness/tape continuity
scripts/verify.sh # read-only verification suite
scripts/sample_ckpt.py chkpt/10k.pt --n 3 --max-tokens 2048 --top-k 40
Limitations
- Small capacity: dream-logic, entity swaps, and invented plots at this scale
- Trained on 19th/early-20th-century children's literature β the style and vocabulary are of that era
- English only, fixed 1,024-token vocabulary
- Repeat loops are a known failure mode at low temperature; sample with top-k