Instructions to use ShayonSarker/SmolLM2-360M-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ShayonSarker/SmolLM2-360M-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ShayonSarker/SmolLM2-360M-GGUF")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("ShayonSarker/SmolLM2-360M-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use ShayonSarker/SmolLM2-360M-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ShayonSarker/SmolLM2-360M-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ShayonSarker/SmolLM2-360M-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ShayonSarker/SmolLM2-360M-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ShayonSarker/SmolLM2-360M-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf ShayonSarker/SmolLM2-360M-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf ShayonSarker/SmolLM2-360M-GGUF:Q4_K_M

Use Docker

docker model run hf.co/ShayonSarker/SmolLM2-360M-GGUF:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use ShayonSarker/SmolLM2-360M-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ShayonSarker/SmolLM2-360M-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ShayonSarker/SmolLM2-360M-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ShayonSarker/SmolLM2-360M-GGUF:Q4_K_M

SGLang

How to use ShayonSarker/SmolLM2-360M-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ShayonSarker/SmolLM2-360M-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ShayonSarker/SmolLM2-360M-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ShayonSarker/SmolLM2-360M-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ShayonSarker/SmolLM2-360M-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use ShayonSarker/SmolLM2-360M-GGUF with Ollama:
```
ollama run hf.co/ShayonSarker/SmolLM2-360M-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use ShayonSarker/SmolLM2-360M-GGUF with Docker Model Runner:
```
docker model run hf.co/ShayonSarker/SmolLM2-360M-GGUF:Q4_K_M
```

Lemonade

How to use ShayonSarker/SmolLM2-360M-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ShayonSarker/SmolLM2-360M-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.SmolLM2-360M-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

SmolLM2-360M GGUF

Source model · HF release · Build hub

Pinned, reproducible llama.cpp GGUF conversion of the 362M-parameter SmolLM2 base model. The source is Apache-2.0 licensed and uses an 8,192-token context window.

Verified outputs

Verbatim smollm2-360m-Q4_K_M.gguf completions, --temp 0, 24 new tokens, prompt form Question: ...\nAnswer:. SmolLM2 answers tersely, then rolls straight into another fabricated Question:/Answer: pair; the extra pairs are trimmed with ….

Question	Model answer
What is the capital of Japan?	`Tokyo`
What is the capital of Italy?	`Rome`
What is the capital of Egypt?	`Cairo`
What is the largest ocean on Earth?	`The largest ocean on Earth is the Pacific Ocean.`
Which planet is closest to the Sun?	`Mercury`
How many days are in a leap year?	`There are 366 days in a leap year.`
How many continents are there?	`There are 7 continents.`
What is the chemical symbol for gold?	`Au`

Eight of eight are correct. Versus SmolLM-360M the answers are shorter and cleaner, and perplexity is lower at every quantization (12.94 vs 14.57 at F16), but the model remains a 362M base LM with no instruction tuning.

Formats

File	Status	Purpose
`smollm2-360m-F16.gguf`	Published	Reference quality
`smollm2-360m-Q8_0.gguf`	Published	Higher-quality compact format
`smollm2-360m-Q4_K_M.gguf`	Published	Smallest release format

Validation

WikiText-2 raw test evaluation, 8 chunks of 512 tokens. Lower perplexity is better.

Format	PPL	Ratio to F16
F16	12.9369	Baseline
Q8_0	12.9416	1.0004
Q4_K_M	13.2124	1.0213

This is a base language model, not an instruction-tuned assistant.

Build

python -m pip install -r requirements-build.txt
python build_gguf.py --model-id HuggingFaceTB/SmolLM2-360M

The builder pins llama.cpp commit 6b790a9c291b5d7af3312bbf9f0c558aa023b13e and the upstream model revision. It does not upload or overwrite this repository.

License

Apache-2.0. See the upstream model card and LICENSE for source attribution.

Downloads last month: 97

GGUF

Model size

0.4B params

Architecture

llama

Hardware compatibility

4-bit

8-bit

16-bit

Model tree for ShayonSarker/SmolLM2-360M-GGUF

Base model

HuggingFaceTB/SmolLM2-360M

Quantized

(44)

this model