CodeBharat-100M

CodeBharat-100M is a 100.68M parameter decoder-only Transformer pretrained from scratch on code (Python, JavaScript, TypeScript, C++, SQL, HTML/CSS, and synthetic textbooks).

Model Details

  • Architecture: Decoder-Only Transformer (Llama/Qwen-style: RMSNorm, RoPE, SwiGLU, Grouped-Query Attention)
  • Parameters: 100,679,424 (100.68M)
  • Vocabulary: 49,152 tokens (Byte-level BPE)
  • Context Window: 1,024 tokens
  • Training Device: NVIDIA GeForce RTX 5050 GPU
  • Final Validation Loss: 1.3891

Quickstart Usage

Native PyTorch / Tokenizers Usage

import torch
from tokenizers import Tokenizer
from pathlib import Path

# Load tokenizer and model weights
tokenizer = Tokenizer.from_file("tokenizer.json")
weights = torch.load("pytorch_model.bin", map_location="cuda" if torch.cuda.is_available() else "cpu")

Run Inference via CLI

python 100m-codebharat/scripts/11_generate.py --prompt "def binary_search(arr, target):"

Model Card Info

  • Developed by: CodeBharat Team
  • Model Type: Causal Language Model for Code
  • License: Apache 2.0
Downloads last month
-
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support