Qwen3.5-4B-Base-Kazakh-OptimizedTokenizer

An optimized new version of ISSAI's Kazakh base model.

Developed by Almubdi Mutaikhan with Batyr Arystanbekov (ISSAI), under the supervision of Prof. Huseyin Atakan Varol (ISSAI). See Citation.

ISSAI earlier released issai/Qwen3.5-4B-Base-Kazakh: Qwen/Qwen3.5-4B-Base with 16,000 added Kazakh tokens and continued pretraining on ~15.7B tokens of Kazakh and English (50 / 50). It made Qwen3.5 much better and much cheaper in Kazakh.

This model is its next version. It keeps the same recipe and changes two things:

  1. A new tokenizer, learned on Kazakh and Russian on top of Qwen's own merges. Kazakh text needs 1.98ร— fewer tokens than with Qwen3.5. Russian needs 1.10ร— fewer (the earlier tokenizer needed 18% more than Qwen3.5). English is tokenized exactly as by Qwen3.5.
  2. Russian added to the training data: 8.4B tokens of Kazakh / Russian / English (40 / 20 / 40).

Result: the best Kazakh and Russian scores of the three models, with English unchanged. The numbers are in Comparison. How it was built is in Implementation.

This is a base model: it continues text and does not follow chat instructions. Use it for fine-tuning or as the starting point for chat models. The chat version (thinking + images) built on it is issai/Qwen3.5-4B-Kazakh-OptimizedTokenizer.

Contents: Model details ยท Implementation ยท Comparison ยท How to use ยท Limitations ยท Citation ยท Contributions

Model details

architecture Qwen3.5: 32 layers, hybrid (24 Gated DeltaNet linear-attention + 8 full-attention layers), hidden size 2,560, text only
parameters 4.25 B
vocabulary 264,077 tokens: Qwen3.5's 248,077 + 16,000 new (ids 248,077โ€“264,076); embedding matrix padded to 264,192 rows
input / output embeddings tied
context length used in training 32,768 tokens
precision bfloat16
languages Kazakh, Russian, English

Implementation

Qwen3.5-4B-Base
  โ”‚ 1. tokenizer: +16,000 Kazakh/Russian tokens (continued BPE)
  โ”‚ 2. new embeddings: mean of the old pieces (FVT)
  โ”‚ 3. phase 1: train the new embedding rows only
  โ”‚ 4. phase 2: train all weights on 8.4B tokens (kk / ru / en 40 / 20 / 40)
  โ–ผ
Qwen3.5-4B-Base-Kazakh-OptimizedTokenizer

1. Tokenizer

Continued BPE: 16,000 new merges were learned on Kazakh and Russian text, appended after Qwen3.5's merge list. Every original token keeps its id and every original merge rule stays, so:

  • English and code are tokenized exactly as by Qwen3.5 (identical on all 488 Belebele passages);
  • Kazakh and Russian words that Qwen split into many small pieces now become one or a few tokens.

Example: าšะฐะทะฐา›ัั‚ะฐะฝ ะ ะตัะฟัƒะฑะปะธะบะฐัั‹ะฝั‹าฃ ะฐัั‚ะฐะฝะฐัั‹ โ†’ 3 tokens (าšะฐะทะฐา›ัั‚ะฐะฝ ยท ะ ะตัะฟัƒะฑะปะธะบะฐัั‹ะฝั‹าฃ ยท ะฐัั‚ะฐะฝะฐัั‹) instead of 10 with Qwen3.5.

2. New embeddings

Each new token's embedding starts as the mean of the embeddings of the Qwen tokens it replaces (Fast Vocabulary Transfer, FVT). Input and output embeddings are tied, so this one table serves both.

3. Phase 1: new embeddings only

Only the 16,000 new embedding rows are trained; all original weights are frozen. The new tokens learn to fit the existing model before anything else changes. 571 steps, ~16,000 sequences of 32,768 tokens, learning rate 1e-4.

4. Phase 2: all weights

One epoch over the 8.4B-token mix: 2,134 steps of 120 sequences ร— 32,768 tokens, learning rate 1e-5 with cosine decay to 1e-6 (2% warm-up), AdamW (ฮฒ = 0.9 / 0.95), weight decay 0.1, 15 ร— H100, DeepSpeed ZeRO-2, ms-swift.

Training data

By tokens: 40% Kazakh, 20% Russian, 40% English; deduplicated.

  • Kazakh: FineWeb-2, FinePDFs, FineWiki, and a multidomain Kazakh corpus (books, news, web).
  • Russian: FineWeb-2, FinePDFs, FineWiki.
  • English (replay, so the model keeps its English): FineWeb-Edu, Nemotron-CC-Math, Stack-Edu and others.

The training data is not released. Each source has its own license.

Comparison

Three models: the original Qwen3.5-4B-Base, ISSAI's earlier Qwen3.5-4B-Base-Kazakh, and this model.

At a glance

Qwen3.5-4B-Base issai/Qwen3.5-4B-Base-Kazakh this model
added tokens โ€” 16,000, Kazakh 16,000, Kazakh + Russian
continued pretraining โ€” ~15.7B tokens, kk / en 50 / 50 8.4B tokens, kk / ru / en 40 / 20 / 40
Kazakh average (6 tasks) 57.37 60.83 61.47
Russian average (4 tasks) 65.37 64.43 65.55
English average (4 tasks) 77.15 76.80 77.01

Tokenizer efficiency

Tokens per word on the 488 Belebele (FLORES) passages of each language. Lower is better: the model works faster, and more text fits in its context.

language Qwen3.5-4B-Base issai/Qwen3.5-4B-Base-Kazakh this model
Kazakh 3.48 1.83 (1.90ร— fewer) 1.76 (1.98ร— fewer)
Russian 1.89 2.23 (1.18ร— more) 1.72 (1.10ร— fewer)
English 1.25 1.25 1.25

Example (Russian): ะ‘ะตัะบั€ะฐะนะฝัั ะบะฐะทะฐั…ัะบะฐั ัั‚ะตะฟัŒ ั‚ัะฝะตั‚ัั ะดะพ ัะฐะผะพะณะพ ะณะพั€ะธะทะพะฝั‚ะฐ, ะธ ะฒ ั‚ะธั…ะพะผ ะฐัƒะปะต ัะปั‹ัˆะฝะพ, ะบะฐะบ ะฟะพั‘ั‚ ะดะพะผะฑั€ะฐ. is 31 tokens with Qwen3.5, 34 with the earlier ISSAI tokenizer, and 29 with this one.

Benchmarks

lm-evaluation-harness, 5-shot, full test sets, bfloat16, the same settings for all three models. Kazakh and Russian tasks are ISSAI's Kazakh/Russian versions of the benchmarks. Accuracy in % (GSM8K: exact match). ยฑ is the standard error.

task Qwen3.5-4B-Base issai/Qwen3.5-4B-Base-Kazakh this model
MMLU (kk) 57.49 ยฑ 0.4 61.88 ยฑ 0.4 61.92 ยฑ 0.4
MMLU-Pro (kk) 35.27 ยฑ 0.4 37.11 ยฑ 0.4 37.43 ยฑ 0.4
MMLU-Redux (kk) 59.07 ยฑ 0.7 63.21 ยฑ 0.6 64.14 ยฑ 0.6
GPQA (kk) 31.38 ยฑ 1.3 34.98 ยฑ 1.4 35.99 ยฑ 1.4
ARC (kk) 85.01 ยฑ 0.6 89.80 ยฑ 0.5 89.77 ยฑ 0.5
Belebele (kk) 76.00 ยฑ 1.4 78.00 ยฑ 1.4 79.56 ยฑ 1.3
Kazakh average 57.37 60.83 61.47
MMLU-Pro (ru) 43.23 ยฑ 0.5 41.65 ยฑ 0.4 43.24 ยฑ 0.5
GPQA (ru) 35.15 ยฑ 1.4 35.15 ยฑ 1.4 37.16 ยฑ 1.4
ARC (ru) 94.53 ยฑ 0.4 93.04 ยฑ 0.4 93.91 ยฑ 0.4
Belebele (ru) 88.56 ยฑ 1.1 87.89 ยฑ 1.1 87.89 ยฑ 1.1
Russian average 65.37 64.43 65.55
MMLU (en) 73.07 ยฑ 0.4 74.28 ยฑ 0.4 73.81 ยฑ 0.4
ARC-Challenge (en) 61.69 ยฑ 1.4 60.92 ยฑ 1.4 62.46 ยฑ 1.4
GSM8K (en) 80.74 ยฑ 1.1 80.44 ยฑ 1.1 79.68 ยฑ 1.1
Belebele (en) 93.11 ยฑ 0.8 91.56 ยฑ 0.9 92.11 ยฑ 0.9
English average 77.15 76.80 77.01

What it shows

  • Kazakh: +4.1 points over Qwen3.5-4B-Base on average, and slightly ahead of the earlier ISSAI model, with a tokenizer that is a little more efficient still.
  • Russian: the main gain of this version. Russian now has its own tokens and training data. It takes 23% fewer tokens than with the earlier tokenizer, and its average is back at Qwen3.5's level (65.55 vs 65.37).
  • English: unchanged within the standard error.

How to use

Requires transformers 5.x (tested with 5.12.1).

from transformers import AutoTokenizer, AutoModelForCausalLM

name = "issai/Qwen3.5-4B-Base-Kazakh-OptimizedTokenizer"
tok = AutoTokenizer.from_pretrained(name)
model = AutoModelForCausalLM.from_pretrained(name, dtype="bfloat16", device_map="cuda")

ids = tok("าšะฐะทะฐา›ัั‚ะฐะฝ ะ ะตัะฟัƒะฑะปะธะบะฐัั‹ะฝั‹าฃ ะฐัั‚ะฐะฝะฐัั‹", return_tensors="pt").to(model.device)
out = model.generate(**ids, max_new_tokens=60)
print(tok.decode(out[0], skip_special_tokens=True))

Limitations

  • A base model: no instruction following, no chat alignment, no safety tuning.
  • Trained at 32,768 tokens per sequence. The architecture allows up to 262,144 (inherited from Qwen3.5), but lengths beyond 32,768 were not evaluated for this model.
  • Its knowledge comes from public web, PDF, wiki, book and news text and has the biases and errors of those sources.
  • Benchmarks are multiple-choice and short-answer tests. They do not measure long-form Kazakh writing quality.

Contributions

  • Almubdi Mutaikhan: tokenizer, training data mix, continued pretraining, evaluation, release.
  • Batyr Arystanbekov: ISSAI's continued-pretraining pipeline and Kazakh training corpus.
  • Huseyin Atakan Varol: supervision.

License

Apache-2.0, the same as Qwen/Qwen3.5-4B-Base.

Citation

@misc{mutaikhan2026qwen35basekazakhopt,
  title        = {Qwen3.5-4B-Base-Kazakh-OptimizedTokenizer},
  author       = {Mutaikhan, Almubdi and Arystanbekov, Batyr and Varol, Huseyin Atakan},
  year         = {2026},
  publisher    = {Hugging Face},
  howpublished = {\url{https://huggingface.co/issai/Qwen3.5-4B-Base-Kazakh-OptimizedTokenizer}}
}
Downloads last month
-
Safetensors
Model size
4B params
Tensor type
BF16
ยท
Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for issai/Qwen3.5-4B-Base-Kazakh-OptimizedTokenizer

Finetuned
(309)
this model
Merges
1 model