Instructions to use ShayonSarker/SmolLM-360M-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ShayonSarker/SmolLM-360M-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ShayonSarker/SmolLM-360M-GGUF")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("ShayonSarker/SmolLM-360M-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use ShayonSarker/SmolLM-360M-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ShayonSarker/SmolLM-360M-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ShayonSarker/SmolLM-360M-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ShayonSarker/SmolLM-360M-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf ShayonSarker/SmolLM-360M-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf ShayonSarker/SmolLM-360M-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama cli -hf ShayonSarker/SmolLM-360M-GGUF:Q4_K_M

Use Docker

docker model run hf.co/ShayonSarker/SmolLM-360M-GGUF:Q4_K_M

Llama new
LM Studio
Jan

vLLM

How to use ShayonSarker/SmolLM-360M-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ShayonSarker/SmolLM-360M-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ShayonSarker/SmolLM-360M-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ShayonSarker/SmolLM-360M-GGUF:Q4_K_M

SGLang

How to use ShayonSarker/SmolLM-360M-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ShayonSarker/SmolLM-360M-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ShayonSarker/SmolLM-360M-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ShayonSarker/SmolLM-360M-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ShayonSarker/SmolLM-360M-GGUF",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Ollama
How to use ShayonSarker/SmolLM-360M-GGUF with Ollama:
```
ollama run hf.co/ShayonSarker/SmolLM-360M-GGUF:Q4_K_M
```
Unsloth Desktop
Docker Model Runner
How to use ShayonSarker/SmolLM-360M-GGUF with Docker Model Runner:
```
docker model run hf.co/ShayonSarker/SmolLM-360M-GGUF:Q4_K_M
```

Lemonade

How to use ShayonSarker/SmolLM-360M-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ShayonSarker/SmolLM-360M-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.SmolLM-360M-GGUF-Q4_K_M

List all available models

lemonade list

Atomic Chat

SmolLM-360M GGUF

Source model · HF release · Build hub

Pinned, reproducible llama.cpp GGUF conversion of the 362M-parameter SmolLM base model. The source is Apache-2.0 licensed and uses a 2,048-token context window.

Formats

File	Status	Purpose
`smollm-360m-F16.gguf`	Published	Reference quality
`smollm-360m-Q8_0.gguf`	Published	Higher-quality compact format
`smollm-360m-Q4_K_M.gguf`	Published	Smallest release format

Verified outputs

Verbatim smollm-360m-Q4_K_M.gguf completions, --temp 0, 24 new tokens, prompt form Question: ...\nAnswer:. Text after the first sentence is trimmed with …. The model is a base LM, so it keeps inventing new questions after answering; that behaviour is left unedited.

Question	Model answer
What is the capital of Japan?	`Tokyo is the capital of Japan.`
What is the capital of Italy?	`The capital of Italy is Rome.`
What is the capital of Egypt?	`The capital of Egypt is Cairo.`
What is the largest ocean on Earth?	`The largest ocean on Earth is the Pacific Ocean. It covers about 30% of the Earth…`
Which planet is closest to the Sun?	`Mercury. It is the closest planet to the Sun.`
How many days are in a leap year?	`366 days.`
How many continents are there?	`There are seven continents in the world.`
What is the chemical symbol for gold?	`The chemical symbol for gold is Au.`

Eight of eight short factual lookups are correct. Treat longer output as unreliable: the model is small, has no instruction tuning, and drifts into a new fabricated Q:/A: pair once it finishes a sentence.

Validation

WikiText-2 raw test evaluation, 8 chunks of 512 tokens. Lower perplexity is better.

Format	PPL	Ratio to F16
F16	14.5685	Baseline
Q8_0	14.6009	1.0022
Q4_K_M	14.8333	1.0182

Build

python -m pip install -r requirements-build.txt
python build_gguf.py --model-id HuggingFaceTB/SmolLM-360M

The builder pins llama.cpp commit 6b790a9c291b5d7af3312bbf9f0c558aa023b13e and the upstream model revision. It does not upload or overwrite this repository.

License

Apache-2.0. See the upstream model card and LICENSE for source attribution.

Downloads last month: 98

GGUF

Model size

0.4B params

Architecture

llama

Hardware compatibility

4-bit

8-bit

16-bit

Model tree for ShayonSarker/SmolLM-360M-GGUF

Base model

HuggingFaceTB/SmolLM-360M

Quantized

(13)

this model