Koliber-1.3-BeLooped

Koliber-1.3-BeLooped is an experimental Polish decoder-only language model developed by OrisTeam as part of its architecture model zoo and private model research.

The model explores weight-shared recurrent Transformer layers, configurable computational depth (x1/x2/x3), and alternating sliding-window and full causal attention.

Koliber v1.3 is an architectural experiment rather than a direct continuation of the Koliber v1.2 design. Its purpose is to investigate alternative ways of allocating parameters and computational depth within the approximately 125M-parameter model class.

This release contains the base pretrained model, without instruction tuning or preference optimization.

Benchmarks

Koliber 1.3 at its default depth (x2) matches Koliber 1.2 on Polish grammar, and at x2 and x3 it has the lowest WikiText-2 perplexity of the Koliber models. Like every Koliber, it was trained on Polish text only, so on English reasoning and knowledge benchmarks it stays at the level of chance.

English results are given for reference only. No Koliber was trained on English text. The English reasoning and knowledge results are noise: they do not measure the models' ability, and the differences between them carry no meaning. English language modeling (WikiText-2, LAMBADA) does separate the models, but measures them on a language they were not trained on.

Benchmark Baseline
(chance or constant)
Koliber 1.3
x2 (default)
Koliber 1.3
x1
Koliber 1.3
x3
Koliber
1.2
Koliber
1.1-Preview
Koliber
1.0
Polish
MultiBLiMP (Polish), accuracy50.095.794.795.795.694.088.2
English: language modeling (reference)
WikiText-2, byte perplexity โ†“โ€“2.9052.9882.9023.0333.7403.757
LAMBADA (OpenAI), accuracy0.39.58.79.17.07.48.0
English: reasoning and knowledge (reference only: noise level, see note)
ARC-Easy, acc26.626.826.327.329.326.925.3
ARC-Challenge, acc_norm26.522.222.322.323.222.122.3
PIQA, acc_norm50.552.352.052.352.651.952.9
WinoGrande, acc50.449.049.649.450.750.750.3
WSC273, acc50.246.549.548.051.649.849.8
Social IQa, acc33.633.333.533.134.134.234.1
MathQA, acc_norm22.620.720.721.020.621.921.2
MedMCQA, acc32.232.032.232.032.132.232.2
PubMedQA, acc55.256.654.054.633.633.833.8

The highlighted value is the best of the Koliber models in each row; ties are all highlighted. Accuracies are in percent. On WikiText-2, lower is better. HellaSwag has not been run yet.

Polish. MultiBLiMP (Polish) asks the model to prefer the grammatical sentence of a minimal pair. Every Koliber is far above chance (50), and the scores grow from 88.2 (1.0) to 95.7 (1.3 x2 and x3). The standard error is about 0.4 points. The other Polish benchmarks of the suite are not yet available in its runner.

English (reference only). No Koliber had English text in its training mixture, so results near chance were expected on English benchmarks, and that is what they show. The reasoning and knowledge rows are noise and are listed for reference, not for comparison. The baseline is the higher of random choice and a constant answer.

  • Across the nine English reasoning and knowledge benchmarks, only one result is above the baseline: Koliber 1.2 on ARC-Easy, at +2.7 points (a probable difference, not a certain one).
  • Several results are below the baseline:
    • on ARC-Challenge, every model is about 4 points under a constant answer;
    • on PubMedQA, Koliber 1.0-1.2 nearly always give an answer other than the most common one, about 21 points under;
    • on WSC273, Koliber 1.3 x2 is 3.7 points under.
  • On MedMCQA, the models give mostly the same answer to every question.

The highlighted values in these rows are therefore not meaningful wins.

Language modeling does separate the models. On WikiText-2, Koliber 1.3 at x2 and x3 has the lowest byte perplexity. On LAMBADA, every model is clearly above the near-zero baseline, but low in absolute terms.

Recurrent depth. On the benchmarks that separate the models (MultiBLiMP, WikiText-2, LAMBADA), x2 and x3 score better than x1, and x3 adds little over x2. These differences between depths, and those between Koliber versions, have not been tested pairwise.

Protocol

All results come from the benchmark suite of SlayerLab / Fabryka AI (CodeSOTA), run with its runner python -m codesota.runner at commit a295888. Every benchmark uses the suite's full test set at a pinned dataset revision, 0-shot log-likelihood scoring through lm-evaluation-harness 0.4.9.1, and the same settings for every model:

Setting Value
Precision float32 (the BF16 weights are loaded in float32)
Batch size 1
BOS one leading BOS token per request (the suite's default)
Hardware NVIDIA GeForce RTX 5060 Ti (16 GB), Windows 11
Software Python 3.12.2, PyTorch 2.8.0+cu128, Transformers 4.57.6, Datasets 3.6.0, Tokenizers 0.22.2
Model code trust_remote_code (the repository's modeling_koliber.py, reviewed before the runs)

Commands, from benchmarks/ of the suite at that commit:

python -m codesota.runner --benchmarks @small,@medium --model OrisTeam/Koliber-1.3-BeLooped \
  --revision c384ec7ffb1efbe0ddc539478fe6986b7084bd95 --model-id OrisTeam/Koliber-1.3-BeLooped@x2 \
  --trust-remote-code --out runs/

Deviations, each recorded in the run's manifest:

  • x1 and x3 run from a local copy of this repository whose config.json sets recurrence_steps to 1 or 3. The weights and code are unchanged.
  • WikiText-2 for Koliber 1.3 uses a 2048-token window (--max-length 2048). With the configured 131,072 positions, the evaluation's windows exceed 2048 tokens, and the model's code requires flash-attn for those, which is not available on this machine. Koliber 1.0-1.2 use their configured 1536 tokens. Perplexity depends on the window, so the 1.3 and older WikiText-2 values are not strictly comparable.
  • Koliber 1.0-1.2 run from local copies whose tokenizer_config.json names the tokenizer class PreTrainedTokenizerFast instead of TokenizersBackend, which Transformers 4.57.6 does not know. The tokenizer file is unchanged, and the token IDs were checked to be identical.

Validation. Every result in the table comes from a run directory that passes the suite's result validator (python -m codesota.results, all 72 runs behind this table). The validator checks the run's structure, checksums, per-question records, and provenance fields. It does not check model quality. The runner itself was verified against the suite's reference pipeline: on Pythia-1B, it reproduces the November 2026 full run on all 13 benchmarks (verification).

Architecture

Property Configuration
Parameters 125,459,712
Architecture Decoder-only, recurrent Transformer
Unique decoder layers 15
Layer distribution 5 input / 6 recurrent / 4 output
Execution depth x1: 15 / x2: 21 / x3: 27
Hidden size 768
FFN dimension 2048
Attention GQA, 16 query heads / 4 KV heads
Head dimension 64
Q projection dimension 1024
Attention pattern Alternating Full / Sliding Window
Sliding window 512 tokens
Normalization Pre-RMSNorm
Positional encoding RoPE
Activation SwiGLU
Linear biases Disabled
Embeddings 32,768 rows, tied input/output
Tokenizer Aster v1.05, 32,000 tokens
Training sequence length 16,384 tokens
Configured maximum positions 131,072
Training stage Base pretraining

Recurrent execution

The architecture consists of three stages:

Input (5 layers) โ†’ Recurrent (6 layers ร— N) โ†’ Output (4 layers)

The recurrent block reuses the same learned parameters across successive passes. Its execution count can be configured independently of the number of unique model parameters.

Mode Executed layers Full attention Sliding-window attention
x1 15 8 7
x2 21 11 10
x3 27 14 13

The architecture combines local attention with periodic access to the complete preceding sequence. The final decoder layer uses full attention.

These execution modes provide different computational budgets while preserving the same model weights. The benchmarks above give a first comparison of their quality; their computational efficiency has not been measured.

Training

The model was pretrained from scratch using the SlayerLab GoLLeM Polish corpus, with a configured mixture of web-derived and curated Polish-language material.

Property Training configuration
Processed tokens 6,000,017,408
Sequence length 16,384
Effective batch 65,536 tokens/update
Optimizer AdamW
Training precision FP32 weights with BF16 autocast
Recurrence Mixed x1/x2/x3
Data sampling configuration 60% web-derived / 40% curated
Hardware NVIDIA RTX PRO 6000 Blackwell

Training used document packing with explicit document boundaries. The reported token count describes processed training exposure, not the number of unique tokens in the underlying corpus.

Context length

The 16,384-token training sequence was used partly to stress-test the training implementation, extended attention, recurrent execution, and the data pipeline.

The model configuration reserves a maximum positional range of 131,072 tokens for future experimentation.

These values must not be interpreted as validated effective context lengths. The ability to retrieve, preserve, or combine information across long sequences has not been established through dedicated evaluation.

The Koliber line

Koliber remains an experimental line of models around 125M parameters, used to try out architectures, training decisions, and data within a fixed, small budget. The versions differ in architecture and in training exposure, so the benchmarks compare models, not controlled changes. In particular, they do not account for inference FLOPs, which grow with the recurrent depth.

The architecture, training decisions, and experimental observations will be discussed in a separate technical note.

Checkpoints and implementation

The repository includes the final model and intermediate training milestones.

Location Contents
Repository root Final 6B-token checkpoint, BF16
fp32/ Final 6B-token checkpoint, FP32
milestones/1B/ Intermediate checkpoint, BF16
milestones/2B/ Intermediate checkpoint, BF16
milestones/3B/ Intermediate checkpoint, BF16
milestones/4B/ Intermediate checkpoint, BF16
milestones/5B/ Intermediate checkpoint, BF16
training/ Training implementation

The Hugging Face implementation provides configurable recurrent execution through recurrence_steps.

The final FP32 export was checked against the original training checkpoint, including weight integrity and numerical logit comparisons for x1, x2, and x3. The BF16 export was separately checked for loading and numerical consistency with FP32 on short test sequences. The benchmarks above use the BF16 export from the repository root.

These implementation checks are not model-quality benchmarks.

Intended use and limitations

Koliber v1.3-BeLooped is intended for architecture research, language-model experimentation, continued pretraining, and downstream fine-tuning.

It is a base language model, not an instruction-following assistant. It has not undergone instruction tuning, preference optimization, or dedicated safety alignment.

The model may produce incorrect information, repetition, incoherent continuations, or misleading text. Reliable factual behavior and long-context capabilities have not been demonstrated.

This release should be treated as an experimental checkpoint of the Koliber architecture family.

License & Credits

Model weights: CC BY-SA 4.0. Source corpus licensing and original text rights are separate from the model-weight license; users are responsible for checking applicable upstream restrictions.

Data source: SlayerLab GoLLeM Polish corpus, prepared by SlayerLab / Arkadiusz Sล‚ota and contributors.

Development: Aleksander Ogrodzki / OrisTeam.

@misc{ogrodzki2026koliber-13,
  author = {Ogrodzki, Aleksander},
  title = {Koliber-1.3-BeLooped},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/OrisTeam/Koliber-1.3-BeLooped}
}
Downloads last month
16
Safetensors
Model size
0.1B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Collection including OrisTeam/Koliber-1.3-BeLooped