---

base_model: null model_creator: Arko007 model_version: v1.0 model_date: 2025-10 model_card_authors: - Arko007

💠 Zenyx-42M: Where Calm Meets Power

Zenyx-42M is a 42M parameter GPT-2 style decoder-only transformer trained from scratch on high-quality educational web text.

The name "Zenyx" fuses Zen (calm, focused intelligence) and Onyx (strength, power)—embodying an efficient, capable language model.

---

Model Details

Component	Value
Architecture	GPT-2 style decoder-only transformer
Parameters	~42M (41.87M)
Layers	8
Hidden Size	512
Attention Heads	8
Context Length	512 tokens
Vocabulary Size	32,000 (BPE)
Positional Encoding	Learned embeddings

Training Configuration

Setting	Value
Training Data	FineWeb-Edu (streamed, deduped, filtered)
Training Mode	Scratch (no pretrain)
Optimizer	AdamW (lr=3e-4, wd=0.1)
LR Schedule	Warmup (2k) + Cosine decay
Batch Size	16 per device, 8 grad. accumulation
Effective Batch	128
Precision	BFloat16 mixed precision
Hardware	NVIDIA L4 (24GB VRAM)
Training Time	~6 hours
Iterations	100,000
Tokens Processed	~6.55B

---

Performance

Model	Params	Val Loss	Zenyx Advantage
Zenyx-42M	42M	3.08	Baseline
GPT-1	117M	~3.3	+22% better
DistilGPT-2	82M	~3.1	Nearly tied
GPT-2 Small	124M	~2.8	3x more efficient

• Training Loss: 0.398
• Validation Loss: 3.08
• Tokens Processed: ~6.55B

---

Benchmarks (measured 2026-10-14)

Re-evaluated with the same code used for the other Zenyx models. Everything below was scored on a Kaggle T4 in fp32 from the checkpoint in this repository (best_model.pt, iteration 99,000). The model's own forward() only returns the last position's logits, so all-position logits were computed by re-running its layers; they match forward() to 8e-6. The context is 512 tokens, so SciQ prompts are truncated to their last 512 tokens.

Zenyx-Fact1000 (internal)

A frozen 1000-question, four-way factual / common-sense completion benchmark (geography, science, history, language, arithmetic, computing; chance 25%): the model must rank the correct completion above three same-type distractors by log-likelihood (sha256 843416524fb39c5f…; not yet published). The "prior-only" control scores the same candidates after the neutral context "The answer is" — a score near it means the model is merely preferring common words.

Model	accuracy	95% interval	without arithmetic	prior-only control
Zenyx-42M (iteration 99,000)	49.5%	46.4–52.6%	54.4%	29.1%
Zenyx_Base_220M	65.3%	62.3–68.2%	73.2%	29.8%
Zenyx-v2-base @96,000 (85M unique)	45.8%	42.7–48.9%	48.1%	30.7%
Zenyx-MCEN-Delta-Mamba @4,100 (565M, still training)	45.5%	42.4–48.6%	45.9%	29.6%
Zenyx-V3-Base @107,200 (1.58B MoE)	81.1%	78.6–83.4%	80.8%	28.7%

SciQ (n = 1000, 0-shot, chance 25%, standard error about 1.5 points)

Model	acc	acc_norm
Zenyx-42M	66.50%	60.30%
Zenyx-v2-base @96,000	52.10%	47.20%
Zenyx-MCEN-Delta-Mamba @4,100	55.30%	49.80%
Zenyx-V3-Base @107,200 (T4 fp16)	79.90%	72.70%

This 42M model, trained only on FineWeb-Edu, scores above Zenyx-v2-base (85M unique parameters) and the early MCEN checkpoint on these two tests; V2's training mixture is 45% mathematics and 35% code, and MCEN's is about half code and mathematics. Zenyx-V3-Base (1.58B parameters, ~81B tokens) is well ahead of all of them. This is a statement about these tests (educational-style factual recall), not about the models' overall quality.

Language modelling (The Pile) — not comparable

This card deliberately gives no Pile bits-per-byte figure for comparison. The tokenizer uses a Whitespace pre-tokenizer (tokenizer.json), so spaces, newlines and indentation are discarded before tokenization and are not modelled, unknown characters become [UNK] (no byte fallback), and text is not reconstructable from the tokens. Bits per byte measured against the source text would therefore be understated relative to models that must also predict whitespace, and cannot be compared with them. In practice this also means the model cannot produce line breaks or code layout.

Notes on the table in "Performance" above

The validation losses in that table were not measured on a common dataset or tokenizer, so they are indicative only and are not comparable to the benchmark numbers above.

Capabilities

• ✅ Text completion
• ✅ Topic consistency
• ✅ Grammar & syntax
• ✅ Technical vocabulary
• ⚠️ Base model (fine-tuning required for instructions)
• ⚠️ 512 token context

---

Usage

Install dependencies

pip install torch transformers tokenizers

Quick Start Example

import torch
from model import NanoGPT
from config import NanoGPTConfig
from tokenizers import Tokenizer

# Load model
checkpoint = torch.load('best_model.pt', map_location='cpu', weights_only=False)
config = checkpoint['config']
model = NanoGPT(config)
model.load_state_dict(checkpoint['model'])
model.eval()

# Load tokenizer
tokenizer = Tokenizer.from_file('tokenizer.json')

def generate(prompt, max_tokens=100, temperature=0.6):
    tokens = tokenizer.encode(prompt).ids
    x = torch.tensor(tokens).unsqueeze(0)
    with torch.no_grad():
        output = model.generate(x, max_new_tokens=max_tokens, temperature=temperature, top_k=40)
    return tokenizer.decode(output.tolist())

prompt = "Artificial intelligence is"
output = generate(prompt)
print(output)

---

Limitations

• ⚠️ Base model only (no instruction fine-tuning)
• ⚠️ Can repeat phrases (use repetition penalty)
• ⚠️ 512 token context window
• ⚠️ Small parameter count limits knowledge
• ⚠️ Possible factual errors
• ⚠️ Not suitable for reliable code generation

Not recommended for:

• ❌ Production applications (without fine-tuning)
• ❌ Factual QA
• ❌ Instruction following
• ❌ Mission-critical tasks

---

Fine-Tuning

Designed as a base model for further fine-tuning.

Next Steps:

1. Instruction fine-tuning (Alpaca/Dolly datasets)
2. Code specialization (GitHub/StackOverflow)
3. Domain adaptation (medical/legal/etc)
4. RLHF (human feedback alignment)

---

Ethical Considerations

• Training data may contain bias
• Intended for educational/experimental use
• Not suitable for high-stakes decisions
• Fully open source and documented
• No explicit safety fine-tuning

---

Citation

@misc{zenyx_42m_2025,
  author = {Arko007},
  title = {Zenyx-42M: Efficient Language Model Trained From Scratch},
  year = {2025},
  month = {October},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Arko007/Zenyx-42M}
}

---

Model Card Authors

• Developer: Arko007
• Date: October 2025
• Contact: HuggingFace Profile

---

License

Apache License 2.0
See LICENSE file for details.

---

Acknowledgments

• HuggingFace FineWeb-Edu team (dataset)
• GPT-2 (OpenAI, architecture inspiration)
• PyTorch (framework)
• NVIDIA L4 GPU (compute)

---

Version History

• v1.0 (Oct 2025): Initial release — 42M params, 100k iterations, val loss 3.08

---

💠 Branding

ZENYX
Symbol: 💠 diamond/geometric
Colors: Deep purple (#5B21B6) + Cyan (#06B6D4)
Font: Modern, clean, geometric
Taglines:

• "Where Calm Meets Power" 💪
• "Efficient Intelligence, Powerful Results" ⚡
• "Small Model, Big Impact" 💎
• "Zen Precision, Onyx Strength" 🔷
  Brand Personality:
• Calm & efficient
• Powerful
• Modern
• Balanced (Zen meets power)

---

Built with 💠 using PyTorch and L4 GPU