Instructions to use Nanbeige/Nanbeige4.2-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Nanbeige/Nanbeige4.2-3B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="Nanbeige/Nanbeige4.2-3B", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Nanbeige/Nanbeige4.2-3B", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use Nanbeige/Nanbeige4.2-3B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "Nanbeige/Nanbeige4.2-3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.2-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/Nanbeige/Nanbeige4.2-3B

SGLang

How to use Nanbeige/Nanbeige4.2-3B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "Nanbeige/Nanbeige4.2-3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.2-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "Nanbeige/Nanbeige4.2-3B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "Nanbeige/Nanbeige4.2-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use Nanbeige/Nanbeige4.2-3B with Docker Model Runner:
```
docker model run hf.co/Nanbeige/Nanbeige4.2-3B
```

Why is it much slower and much more memory hungry than slightly larger models such as Qwen3.5-4B?

#31

by slightlyoutofphase - opened Aug 13

Discussion

slightlyoutofphase

Aug 13

Definitely not what I was expecting.

leran1995

Nanbeige LLM Lab org Aug 16

Thank you for raising this. The Nanbeige4 series (Nanbeige4, Nanbeige4.1, Nanbeige4.2, and the upcoming Nanbeige4.5) focuses on improving model quality under a fixed parameter budget. In Nanbeige4.2, the Looped Transformer and disabled KV sharing improve model performance but result in slower decoding.

We are preparing a DFlash version of Nanbeige4.2 for faster inference. Nanbeige5 will use linear attention to further improve inference efficiency.

iloveducks

Aug 16

What's the ETA for Nanbeige 4.5?

aliq96

Aug 16

What's the ETA for Nanbeige 4.5?

https://huggingface.co/Nanbeige/Nanbeige4.2-3B/discussions/29

zeerd

Aug 20

Waiting for this DFlash . It's too slow for me right now. Just a half of gemma-4-e4b on my side.

coder543

21 days ago

https://huggingface.co/Nanbeige/Nanbeige4.2-3B-DSpark

Ohjaaja

13 days ago

Best small model for small coding i've tested. However on my dear macbook air m4 16gb it's slooooow. 3-10t/s. I am sure there is some metal acceleration missing even tho llama.cpp server output looks fine. On my rtx3060 12gb pc it does 45-55 t/s.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment