kipasyangin5/5m-terminal-lm-chinchilla

5M Terminal & Multilingual Language Model (Chinchilla Optimal - 100M Tokens)

This repository contains a 5.0 Million Parameter Causal Language Model trained from scratch on a compute-optimal token budget of 100 Million Tokens (20x parameter count) following Chinchilla scaling laws.

The model is specialized in Linux Terminal Commands, Shell Automation, and Multilingual Text Generation (English, Indonesian, Spanish, French, German).

---

📊 Dataset Breakdown & Raw Corpus Links

The model was trained on 100,000,000 Tokens mixed across three distinct sources:

Dataset Name	Source / Hugging Face ID	Token Count	Share (%)	Utilization Details
Linux Terminal & CLI Engine	kipasyangin5/terminal-cli-commands-dataset	~20.0 Million	20%	100% Fully Utilized: Pure synthesized Linux CLI commands (cd .., ls -la, mkdir, grep, git, chmod, curl, Q&A pairs).
English General Text	wikitext (wikitext-2-raw-v1)	~24.0 Million	24%	Streamed Sample: Streamed line-by-line via Hugging Face load_dataset("wikitext", streaming=True).
Multilingual Wikipedia	wikimedia/wikipedia	~56.0 Million	56%	Streamed Multilingual Sample: Streamed across 4 language sub-splits (Indonesian id, Spanish es, French fr, German de).
TOTAL	Mixed Corpus	100.0 Million	100%	Chinchilla Compute-Optimal (20x params)

🔗 Direct Dataset Downloads:

• 💻 Pure CLI Commands Dataset: huggingface.co/datasets/kipasyangin5/terminal-cli-commands-dataset
• 📦 Full 50M Token Mixed Corpus: huggingface.co/datasets/kipasyangin5/5m-terminal-lm-dataset

---

📂 Complete Training Source Code

The full, self-contained training codebase is included directly in this repository inside the /training_code folder:

• training_code/main_kaggle.py: Complete Kaggle GPU standalone training script.
• training_code/train.py: Main PyTorch model architecture, loss functions, and HF auto-upload.
• training_code/terminal_dataset.py: CLI command generator and shell interaction builder.
• training_code/tokenizer_builder.py: Custom 4,096 BPE Tokenizer trainer.

---

⚙️ Model Architecture

• Total Trainable Parameters: 4,984,064 (~4.98M / 5.0M)
• Architecture: Decoder-only Transformer (LLaMA style with SwiGLU & RMSNorm)
• Vocabulary Size: 4,096 (Byte-Pair Encoding Tokenizer)
• Hidden Dimension (d_model): 256
• Number of Layers (n_layer): 6
• Number of Attention Heads (n_head): 8 (Head dimension = 32)
• MLP Hidden Dimension (inter_dim): 512 (SwiGLU activation)
• Context Window (max_seq_len): 256 tokens
• Weight Tying: Tied Embedding and LM Head weights

---

💻 Usage & Inference Example (PyTorch)

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "kipasyangin5/5m-terminal-lm-chinchilla"
model = AutoModelForCausalLM.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

inputs = tokenizer("User: How do I navigate up one directory?\nAssistant:", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=30)
print(tokenizer.decode(outputs[0]))

Citation & License

MIT License. Developed by kipasyangin5.