SOVYTHOS-V2

🦋 SOVYTHOS-V2

Sovythos Egyptian Reasoning & Code Base Model

A production-grade, Llama-compatible decoder-only Transformer trained entirely from scratch for Arabic, English, and Code.

⚠️ Training Status: Early-Stage Checkpoint

This checkpoint is an early snapshot from an ongoing training run — it is NOT the final SOVYTHOS-V2 release. The model is uploaded at this stage for architecture testing, integration testing (transformers/GGUF compatibility), and community feedback purposes only.

Expect at this stage:

Weak or incoherent generations on many prompts

Little to no instruction-following ability (this is a Base, non-instruction-tuned model on top of being early-stage)

Validation loss still actively decreasing — quality will improve substantially with further training

Training is actively continuing. Follow this repository for updates as later checkpoints are released.

🦋 Overview

SOVYTHOS-V2 is the second-generation foundation model of the Sovythos AI Project — a fully open-source, decoder-only Transformer trained from randomly initialized weights on a multilingual corpus spanning Modern Standard Arabic, Egyptian Arabic, English, and programming languages.

This release represents a significant architectural leap from the original 66M Base model, scaling to ~300 million parameters while adopting a Llama-compatible design (RMSNorm, RoPE, GQA, SwiGLU) to ensure seamless deployment across the modern inference ecosystem, including llama.cpp, llama-cpp-python, and Hugging Face transformers.

SOVYTHOS-V2 is released as a Base (Pretrained) model. It has not undergone instruction tuning or RLHF and is intended for researchers, developers, and educators who require a sovythos, transparent foundation for downstream training.

Key Design Principle: Every component — from the tokenizer to the tensor naming convention — is engineered for direct compatibility with the Llama/GGUF toolchain. No conversion hacks required.

📊 Model Specifications

Property	Value
Model	SOVYTHOS-V2
Parameters	~300 Million
Architecture	Decoder-only Transformer (Llama-style)
Hidden Size	1,024
Layers	24
Attention Heads	16 (Query) / 4 (Key-Value)
Head Dimension	64
Context Length	2,048 tokens
RoPE Theta	1,000,000
Vocabulary Size	32,000
Framework	PyTorch 2.x
Weight Format	Safetensors + GGUF (F16 / F32)
Training	From Scratch
Release	Base

🏗️ Architecture Details

SOVYTHOS-V2 implements a modern, efficient decoder stack aligned with the Llama architecture specification for frictionless ecosystem integration.

Normalization

RMSNorm applied pre-attention and post-attention (no bias, no learned affine shift)

Position Encoding

RoPE (Rotary Position Embedding) with rotate_half (NEOX-style) implementation
Base frequency θ = 1,000,000
Precomputed cosine/sine caches up to max sequence length

Attention Mechanism

Grouped Query Attention (GQA) with 16 query heads and 4 key-value heads
Reduces KV-cache memory footprint by 4× during inference compared to standard MHA
Compatible with memory-efficient attention kernels (scaled_dot_product_attention)

Feed-Forward Network

SwiGLU activation with gated projection
Hidden dimension rounded to nearest multiple of 256 for optimal GPU kernel utilization
gate_proj, up_proj, down_proj naming matches Llama/GGUF conventions exactly

Weight Initialization

Gaussian initialization (σ = 0.02) for embeddings and linear layers
Scaled down-projection and output-projection initialization by 1/√(2×layers) to stabilize deep training

Training Optimizations

torch.compile graph mode (when available)
TF32 matmul acceleration on Ampere+
Automatic Mixed Precision (AMP) with fp16 forward pass
Gradient Checkpointing for long-context training on consumer GPUs
Gradient Accumulation with effective batch scaling
Cosine LR Decay with linear warmup
Gradient Clipping (max norm = 1.0)

🔤 Tokenizer

SOVYTHOS-V2 ships with a custom-trained Byte-Pair Encoding (BPE) tokenizer optimized for Arabic morphology, English, and code syntax.

Design Choices

Feature	Implementation
Pre-tokenization	GPT-4 / cl100k-style regex split (Unicode-aware)
Numerical Splitting	Digits grouped by 3 (e.g., `1,234,567` → `1` `234` `567`)
Fallback	ByteLevel encoding for lossless coverage of any Unicode symbol
Normalization	NFC Unicode normalization (unifies Arabic hamza & diacritic forms)
Special Tokens	`<\|endoftext\|>`, `<\|pad\|>`, `<\|user\|>`, `<\|assistant\|>`, `<\|system\|>`, `<\|code\|>`, `<\|/code\|>`, `<\|think\|>`, `<\|/think\|>`
Vocab Size	32,000
Min Frequency	2

Coverage Efficiency (Tokens per Character)

Lower is better. Measured on a held-out sample of the training corpus:

Domain	Tokens / Char	Notes
Arabic (MSA + Egyptian)	~0.35	Efficient subword decomposition for Semitic morphology
English	~0.28	Competitive with modern English-centric tokenizers
Code	~0.42	Strong performance on Python / JavaScript identifiers

The tokenizer is saved as a standard tokenizer.json (Hugging Face tokenizers format) and its merges/vocab are directly compatible with GGUF gpt2 tokenizer model type.

🏋️ Training

SOVYTHOS-V2 was trained entirely from scratch. No pretrained weights from Llama, GPT, or any other foundation model were used as initialization.

Data Pipeline

Corpus: Multilingual web text, technical documentation, mathematical content, and code repositories
Languages: Modern Standard Arabic, Egyptian Arabic, English, and multilingual programming languages
Cleaning: Duplicate-line removal, short-line filtering (< 2 chars), NFC normalization, and aggressive deduplication of repeated contiguous lines
Format: Binary tokenized memmap for high-throughout training with np.uint16 encoding

Training Recipe

Hyperparameter	Value
Optimizer	AdamW (β₁=0.9, β₂=0.95, ε=1e-8)
Peak Learning Rate	3e-4
Minimum Learning Rate	3e-5
Warmup Steps	1,000
LR Schedule	Cosine decay
Weight Decay	0.1 (2D+ tensors only)
Max Gradient Norm	1.0
Precision	FP16 (AMP) with FP32 master weights
Batch Size	4 (per device)
Gradient Accumulation	8 steps
Effective Batch Size	32
Block Size	512 tokens

Checkpointing & Resumption

Checkpoints saved in Safetensors format (secure, fast, language-agnostic)
Automatic resumption from ckpt_dir/best/ including step count and best validation loss
Best checkpoint selection based on validation loss

Export

Native GGUF export after training (or from existing checkpoints via --export_gguf_only)
Supported quantizations: F32, F16 (post-hoc Q4/Q8 via llama.cpp tools)
Tensor name map strictly follows Llama GGUF specification (blk.{i}.attn_q.weight, blk.{i}.ffn_gate.weight, etc.)

🚀 Try It Now (Google Colab)

The fastest way to test SOVYTHOS-V2 is directly through 🤗 transformers, using the custom model code shipped in this repository (trust_remote_code=True). No manual setup, no cloning — just run the cell below in a fresh Colab notebook.

# ============================================================
# 🔱 SOVYTHOS-V2 (303M-Base) — Colab Quickstart
# ============================================================

!pip install -q -U transformers accelerate safetensors

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM

MODEL_ID = "my0919175/Sovythos-303M-Base"

print("🔄 Loading tokenizer...")
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)

print("🔄 Loading model...")
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    dtype=torch.float32,
    trust_remote_code=True,
).to("cuda" if torch.cuda.is_available() else "cpu")
model.eval()

# Chat-style prompt using the built-in special tokens
prompt = "<|user|>\nمرحبا, احكيلي عن نفسك\n<|assistant|>\n"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)

with torch.no_grad():
    output = model.generate(
        input_ids,
        max_new_tokens=150,
        do_sample=True,
        temperature=0.8,
        top_k=40,
        top_p=0.9,
        repetition_penalty=1.15,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

Note: This is an early-stage training checkpoint of a Base (non-instruction-tuned) model — not the final release. Outputs may be short, repetitive, incoherent, or diverge from the prompt entirely; this is expected at this stage of training and is not a loading error. Quality will improve as training progresses and with the upcoming Instruct release.

When prompted Do you wish to run the custom code? [y/N], type y — this loads configuration_sovythos.py and modeling_sovythos.py from this repo, which implement the GQA + RoPE + SwiGLU architecture described above.

💻 Usage

Loading with 🤗 Transformers (Recommended)

This repository ships with trust_remote_code-compatible files (configuration_sovythos.py, modeling_sovythos.py), so it works directly with the standard AutoModelForCausalLM / AutoTokenizer API — no need to clone the original training repo:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "my0919175/Sovythos-303M-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.float32, trust_remote_code=True
).to("cuda" if torch.cuda.is_available() else "cpu")

prompt = "<|user|>\nThe capital of Egypt is\n<|assistant|>\n"
input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(model.device)

output = model.generate(
    input_ids,
    max_new_tokens=50,
    temperature=0.8,
    top_k=40,
    top_p=0.9,
    eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

Loading with the Native Training Code

If you have the original training repository (model.py, train.py) available locally, you can also load the checkpoint directly through the native Model class, which supports the custom KV-cache and generation loop used during training/research:

from model import Model

model = Model.from_pretrained("sovythos_ckpt/best", device="cuda")

text = "The capital of Egypt is"
ids = tokenizer.encode(text).ids
input_ids = torch.tensor([ids], device="cuda")

output = model.generate(
    input_ids,
    max_new_tokens=50,
    temperature=0.8,
    top_k=40,
    top_p=0.9,
    eos_token_id=tokenizer.token_to_id("<|endoftext|>")
)
print(tokenizer.decode(output[0].tolist()))

Using the Tokenizer Directly

from tokenizers import Tokenizer

tok = Tokenizer.from_file("tokenizer.json")
encoded = tok.encode("إزيك يا كبير؟")
print(encoded.ids)  # [ ... ]
print(tok.decode(encoded.ids))  # "إزيك يا كبير؟"

🚀 GGUF Inference

Because SOVYTHOS-V2 uses Llama-compatible tensor naming and a GPT-2-format tokenizer, the exported GGUF file runs out-of-the-box with llama.cpp and llama-cpp-python.

llama-cpp-python

from llama_cpp import Llama

llm = Llama(
    model_path="sovythos-v2.f16.gguf",
    n_ctx=2048,
    n_gpu_layers=-1,  # Offload all layers to GPU
    verbose=False
)

output = llm(
    "The capital of Egypt is",
    max_tokens=50,
    temperature=0.8,
    top_k=40,
    top_p=0.9,
    stop=["<|endoftext|>"]
)
print(output["choices"][0]["text"])

llama.cpp CLI

./main -m sovythos-v2.f16.gguf        -n 50        --temp 0.8        --top-k 40        --top-p 0.9        -p "The capital of Egypt is"

No conversion scripts required. The GGUF file is produced directly by the training framework with correct llama arch metadata.

🎯 Intended Uses

SOVYTHOS-V2 is designed as a research and development foundation for:

Continued pretraining on domain-specific corpora
Supervised fine-tuning (SFT) for chat or task-specific behavior
Instruction tuning and alignment experiments
Educational exploration of modern decoder-only architectures
Arabic NLP research (MSA, Egyptian dialect, code-switching)
Low-resource and Sovythos AI development

Not intended for: Direct deployment as a production chatbot without additional safety fine-tuning and evaluation.

⚠️ Limitations

As a Base (pretrained) model, SOVYTHOS-V2 exhibits expected limitations:

No instruction following without fine-tuning
Hallucination of factual information (names, dates, technical details)
Repetition in long-form generation
Incomplete reasoning on complex multi-step problems
Limited context coherence beyond a few hundred tokens in zero-shot settings
Code generation produces syntactically valid structures but may contain logic errors

These limitations are standard for base models of this scale and will be addressed in future Instruct/Chat releases.

🛣️ Roadmap

Milestone	Status
🟡 SOVYTHOS-V2 Base	In Training (early checkpoint published)
⬜ SOVYTHOS-V2 Base (final)	Planned
⬜ SOVYTHOS-V2-Instruct	Planned
⬜ SOVYTHOS Chat	Planned
⬜ Larger Parameter Models (1B+)	Planned
⬜ Long Context Extension (8K+)	Planned
⬜ Advanced Reasoning Tuning	Planned
⬜ Improved Egyptian Dialect Understanding	Planned

🌌 Sovythos AI Project

SOVYTHOS-V2 is the flagship foundation model of the Sovythos AI Project, an open ecosystem dedicated to building transparent, multilingual language models with native Arabic support and modern reasoning capabilities.

The project prioritizes:

Sovereignty: Full training pipeline ownership (data → tokenizer → model → export)
Transparency: Open weights, open code, open training logs
Accessibility: GGUF compatibility for edge and consumer hardware
Community: Research-first releases with clear limitation disclosures

📖 Citation

If you use SOVYTHOS-V2 in your research, please cite:

@misc{sovythosv2,
  title={SOVYTHOS-V2: Sovythos Egyptian Reasoning and Code Base Model},
  author={Mahmoud Yasser},
  year={2026},
  publisher={Hugging Face},
  howpublished={\url{https://huggingface.co/my0919175/Sovythos-303M-Base}}
}

👨‍💻 Creator

Mahmoud Yasser

Founder of the Sovythos AI Project

📄 License

Apache License 2.0

🦋 Welcome to the next chapter of the Sovythos ecosystem.

This is an early, actively-training checkpoint shared in the spirit of open, transparent development — not a finished product. Researchers, developers, and builders who believe in open, sovereign AI are welcome to follow along as training progresses.

Future checkpoints will bring lower loss, better coherence, instruction tuning, and long-context support.

⭐ If you find this project useful, please consider following the repository and sharing your experiments.

Downloads last month: 466

Safetensors

Model size

0.3B params

Tensor type

F32

my0919175
/

Sovythos-303M-Base