Maggio33/gollem-v4-250m-pl

GoLLeM v4 — 250M (Polish, research base model — final checkpoint + training trajectory)

⚠️ STATUS: TRAINING COMPLETE — RESEARCH BASE MODEL. This repository contains the final checkpoint (ckpt_48828, 100% of the planned budget) together with the full series of training snapshots (roughly 4–100% of training), released for research purposes only. This is a base (pretrained) model, not instruction-tuned, and not a production system. It is a 250M research model trained on a single GPU — do not treat it as a general measure of "how good a Polish LM can be".

GoLLeM v4 is a 250M-parameter decoder-only (GPT-style) Polish language model, trained from scratch on a Polish corpus. It is a Slayer Lab project. We publish the full series of snapshots from successive stages of training (through the final checkpoint) so that others can study how the model learns over time.

Author: Arkadiusz Słota (Slayer Lab).

What it is for (and what it is not)

Intended use: research on scaling and training dynamics (comparing successive snapshots), analysis of a Polish language model, and tests of fluency and text continuation in Polish.

Not an instruction-following model. This is a base (pretrained) model, not instruction-tuned: it continues the text you give it, it does not follow commands. It understands Polish only — by design, it performs poorly in English and when asked questions without context. It does not reliably end its output (it does not consistently emit the <|endoftext|> token; at the final checkpoint the measured end-of-sequence rate is ~0, so it generates up to the length limit). This is expected for a base model of this size, not a bug.

Training snapshots (trajectory)

Snapshot	Step	% of training	Validation loss	BPB (held-out)
ckpt_2000	2000	~4%	3.416	1.135
ckpt_4000 (metric only)	4000	~8%	—	1.042
ckpt_6000	6000	~12%	2.966	1.001
ckpt_8000	8000	~16%	2.937	0.978
ckpt_10000	10000	~20%	2.878	0.961
ckpt_12000	12000	~25%	2.846	0.947
ckpt_14000	14000	~29%	2.849	0.938
ckpt_16000	16000	~33%	2.789	0.932
ckpt_18000	18000	~37%	2.799	0.924
ckpt_20000	20000	~41%	2.757	0.915
ckpt_22000	22000	~45%	2.756	0.910
ckpt_24000	24000	~49%	2.689	0.903
ckpt_26000	26000	~53%	2.670	0.901
ckpt_28000	28000	~57%	2.652	0.897
ckpt_30000	30000	~61%	2.672	0.891
ckpt_32000	32000	~66%	2.692	0.888
ckpt_34000	34000	~70%	2.686	0.884
ckpt_36000	36000	~74%	2.639	0.881
ckpt_38000	38000	~78%	2.650	0.877
ckpt_40000	40000	~82%	2.643	0.8746
ckpt_42000	42000	~86%	2.608	0.8731
ckpt_44000 (metric only)	44000	~90%	2.625	0.8712
ckpt_46000	46000	~94%	2.612	0.8700
ckpt_48000	48000	~98%	2.568	0.8696
ckpt_48828 (final)	48828	100%	—	0.8700

Note on metric-only rows (ckpt_4000, ckpt_44000): their held-out BPB was measured and is shown for trajectory continuity, but their weights are not published — ckpt_4000 is an early snapshot that was never saved to the release series, and ckpt_44000 was rotated out during training. Every other listed snapshot is downloadable.

Training is complete: the final checkpoint is ckpt_48828 (48,828 steps; the final uninterrupted segment took ~71 h on a single RTX 5090, see Training recipe). BPB falls steadily across the run (0.978 → 0.870), flattening to a plateau in the final steps; validation loss tracks training loss closely, so there is no sign of overfitting. The small bumps in validation loss are single-batch noise, not a change in trend. Over the final steps (46000–48828) the held-out BPB has converged to a plateau of ~0.8696–0.8700 — the sub-0.001 variation between these points is evaluation noise (16-window eval), not continued improvement; the model has effectively converged at this token budget. (The final checkpoint fell between evaluation points, so it has no separate validation-loss reading; its held-out BPB was measured directly.)

How we validated the forecast (out of sample). Throughout training we tested a power-law forecast on steps that were not part of the fit:

• The prediction for step 18000 was on target. Between steps 20000 and 26000 the model came in below the extrapolation — it learned faster than the power law predicted.
• From step 26000 onward, results matched the refreshed fit within Δ ≈ −0.002 to −0.004 BPB. The forward check held twelve times in a row (steps 26000–48000), through the second half of the cosine learning-rate decay (there was no separate annealing phase; see Training recipe).
• An anti-collapse diversity monitor (distinct-n on generated text) flagged a transient dip at step 38000 (distinct-3 fell to 0.930). This was a benign annealing transient, not memorization: the direct held-out test excluded memorization — held-out BPB kept falling through the flagged window (0.8807 → 0.8772 → 0.8746) — and distinct-n rebounded. Across the whole run the model is over-determined benign on both axes — the direct held-out discriminator (primary) and the distinct-n proxy (secondary, 0.9564 at the endpoint) — with memorization directly disconfirmed throughout.

End of training (measured, not forecast). The endpoint is no longer an estimate. At the final checkpoint the held-out BPB = 0.8700 (bits-per-byte), measured at step 48828 and recompute-verified independently (two separate evaluation runs — the trainer's and an independent re-computation — produced the same number; the evaluation path was validated by reproducing the step-48000 value exactly). Across the deep tail (steps 46000–48828) the held-out BPB plateaus at 0.8696–0.8700, essentially flat within evaluation noise. For the record, the earlier power-law forecast (0.869) was mildly optimistic: the real measurement came in at 0.8700, consistent with the observed flattening. The theoretical floor for the language is around 0.83 BPB.

Architecture

Parameters	247,954,432 (~250M, with tied embeddings)
Layers (n_layer)	17
Model width (d_model)	1024
Attention heads (n_head)	16 (head dim 64)
Context length (block)	1024
Vocabulary	32,000 (V32k tokenizer, Polish BPE)
Positional encoding	learned positional embeddings (nn.Embedding, not RoPE)
Block	pre-LayerNorm, causal SDPA attention, 4× GELU MLP
Weights	input embedding and output head are tied (weight tying, as in GPT-2)

The architecture is custom and does not load via transformers/AutoModel. Use the included loader modeling_gollem.py (standalone; requires only torch, tokenizers and safetensors).

Training data

Corpus: SlayerLab/gollem-corpus-16b-pl — 16.58B tokens, 100% Polish, cleaned (deduplication, PII filtering, decontamination):

Register	Tokens	Share
web (HPLT v3 PL, cleaned)	14.62B	88.2%
encyclopedic (Polish Wikipedia)	984M	5.9%
science	522M	3.1%
news	211M	1.3%
legal	176M	1.1%
literary	60M	0.4%
mixed	1.6M	0.01%

Training target: 24B tokens (~1.45 epochs). This is deliberate overtraining — roughly 5× the Chinchilla-optimal token count — which is optimal for the cost of later inference rather than for training itself. Hardware: a single RTX 5090.

Training recipe

Everything below is taken from the training script and the run log of this model; values the logs do not record are marked as not recorded.

Script: train_250m.py in this repository (a copy of the file with sha256 70d53f682b75752f…; local example paths and internal ticket ids in comments removed, and checkpoints loaded with torch.load(..., weights_only=True) when resuming; no other change). One process on one GPU.

Initialisation	from scratch; GPT-2 style: normal(0, 0.02) for linear and embedding weights, biases 0, residual projections scaled to 0.02 / sqrt(2 × 17)
Optimizer	AdamW, betas (0.9, 0.95), eps 1e-8 (PyTorch default), weight decay 0.1 on all parameters (one group), gradient clipping 1.0
Learning rate	linear warmup for 488 steps (1 % of training) to 3e-4, then one cosine decay to 3e-5 (0.1 × peak) at the last step; no separate annealing phase
Batch	480 sequences × 1,024 tokens = 491,520 tokens per step (24 sequences × 20 gradient-accumulation steps)
Steps / tokens	48,828 steps = 24.0B tokens (about 1.45 passes over the 16.5B-token training split)
Precision	bf16 autocast; weights and optimizer state in fp32
Data sampling	random 1,024-token windows drawn with replacement; each register file is sampled in proportion to its token count (web 0.882, encyclopedic 0.059, science 0.031, news 0.013, legal 0.011, literary 0.004, mixed 0.0001), fixed for the whole run; windows may cross document boundaries
Held-out set	the held-out split of the corpus (84.5M tokens per the corpus card, documents chosen by hash of the document id), used only for validation every 500 steps (16 batches)
Seed	1337
Checkpoints	every 2,000 steps; the release is the last step (48,828), not a selected checkpoint
Hardware / time	one RTX 5090 (32 GB), ~89.5k tokens/s; the final uninterrupted segment (steps 2,000 → 48,828) took 71.4 h

Command (paths replaced by placeholders):

torchrun --standalone --nproc_per_node=1 train_250m.py \
  --data <train register files> --val-data <held-out file> \
  --total-tokens 24e9 --batch 480 --accum-steps 20 \
  --ckpt-dir <ckpt dir> --log-dir <log dir> \
  --ckpt-every 2000 --log-every 50 --val-every 500 --run-id v4-fire

Defaults used by this command: --lr 3e-4, --wd 0.1, --warmup-frac 0.01, --val-batches 16, --seed 1337, --keep-ckpt 3.

What happened during the run (for reproducibility):

• The first attempt crashed twice (SIGBUS) around steps 2,000–3,650. The released run resumed from the step-2,000 checkpoint on the original data and ran without interruption to the end. A short restart on a different data build (steps 2,000–2,300) was discarded and is not part of this model.
• Random-number state is not saved in checkpoints; on resume the seed is set again, so the data-window order after step 2,000 partly repeats the order from the start of training.
• The training split size logged by the trainer (16,514,371,180 tokens) is 19.2M tokens larger than the train count on the corpus card; the reason is not recorded.

Usage

pip install torch tokenizers safetensors
# final checkpoint (default):
python generate.py "Polska to kraj" --n 100 --temp 0.8 --topk 50
# a specific snapshot from the trajectory:
python generate.py "Polska to kraj" --ckpt ckpt_6000.safetensors --n 100

from modeling_gollem import GollemGPT
m = GollemGPT.from_pretrained("./")                          # final checkpoint
# m = GollemGPT.from_pretrained("./", ckpt="ckpt_6000.safetensors")  # a specific stage
print(m.generate("Stolica Polski to", max_new_tokens=100, temperature=0.8, top_k=50))

Files in the repository: ckpt_*.safetensors (weights of each snapshot, pickle-free format), config.json (architecture and list of snapshots), tokenizer.json (V32k), modeling_gollem.py and generate.py (standalone loader and generation script), train_250m.py (training script, see Training recipe).

Note on format: weights are stored as safetensors (pure tensors, no executable code), which is safer for public release than raw .pt (pickle) files. Full training checkpoints (with optimizer state, for resuming training) remain internal.

Limitations and ethical notes

• This is a 250M research base model; although training is complete, it is small and is not a production or instruction-following system.
• It was trained on web data; despite filtering, it may reproduce biases, errors and content present in the corpus.
• A PII filter was applied (residual rate ≈ 0.0018%), but this does not guarantee the complete absence of personal data.
• Research use only. Slayer Lab accepts no responsibility for production use.

License and data provenance

Weights: Creative Commons Attribution-ShareAlike 4.0 International (CC BY-SA 4.0). You may use, adapt and publish derivatives (including fine-tunes) under the terms of this licence, provided you give attribution (GoLLeM v4, Arkadiusz Słota / Fabryka AI, link to this repository) and release derivative weights under the same licence.

Training data: SlayerLab/gollem-corpus-16b-pl. Partitions default and cc-by-sa contain only apache-2.0, CC-BY, CC-BY-SA and official documents, with no non-commercial records. Partition web (HPLT v3 PL, ~88 % of tokens) is used on the text-and-data-mining basis described on the corpus card; its license column shows CC0, which is the licence of the HPLT compilation, not of the texts. News items carry the dataset licence; the articles remain under their authors' copyright. This is our description of the data, not legal advice.

---

GoLLeM v4 — Slayer Lab. Author: Arkadiusz Słota. Research base model (final checkpoint + full training trajectory); card maintained by the Slayer team.