Nebium

Causal Transformer for self-supervised next-move prediction on chess games.

Architecture

• d_model: 64
• n_heads: 4
• n_layers: 1
• vocab_size: 136
• max_seq_len: 128
• positional_encoding: rope
• activation: swiglu
• norm: rmsnorm

Last validation metrics

• val/loss: 8.162410736083984
• val/accuracy: 0.10000000149011612
• val/top5_accuracy: 0.15000000596046448
• val/perplexity: 3506.629981726336