Instructions to use google/gemma-3-1b-it-qat-int4-unquantized with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use google/gemma-3-1b-it-qat-int4-unquantized with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="google/gemma-3-1b-it-qat-int4-unquantized")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("google/gemma-3-1b-it-qat-int4-unquantized")
model = AutoModelForCausalLM.from_pretrained("google/gemma-3-1b-it-qat-int4-unquantized", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use google/gemma-3-1b-it-qat-int4-unquantized with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "google/gemma-3-1b-it-qat-int4-unquantized"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-3-1b-it-qat-int4-unquantized",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/google/gemma-3-1b-it-qat-int4-unquantized

SGLang

How to use google/gemma-3-1b-it-qat-int4-unquantized with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "google/gemma-3-1b-it-qat-int4-unquantized" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-3-1b-it-qat-int4-unquantized",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "google/gemma-3-1b-it-qat-int4-unquantized" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-3-1b-it-qat-int4-unquantized",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use google/gemma-3-1b-it-qat-int4-unquantized with Docker Model Runner:
```
docker model run hf.co/google/gemma-3-1b-it-qat-int4-unquantized
```

is this per-channel int4 quantized?

by anemll - opened May 12, 2025

Discussion

anemll

May 12, 2025

https://storage.googleapis.com/deepmind-media/gemma/Gemma3Report.pdf
tech report says

" Based on the most popular open source quantization inference engines (e.g. llama.cpp), we focus on three weight representations: per-channel int4, per-block int4, and switched fp8. In Table 3, we report the memory filled by raw...."

Does it mean this is "per-channel int4" described? Q4_0 is clearly block-32 per
this:
https://github.com/ggml-org/llama.cpp/wiki/Tensor-Encoding-Schemes

lkv

Google org Sep 23, 2025

•

edited Sep 23, 2025

Hi Sorry for late reply,

In the context of llama.cpp, Q4_0 is a specific implementation of a per - block quantization scheme. As you correctly noted the from llama.cpp wiki, it uses a block of 32. Therefore, while the Gemma report uses the general term " per-block int4 " , Q4_0 is a concrete example of this technique with a specific block size.

So report is not saying the Gemma use Q4_0. Its saying that is was evaluated with a general " per-block int4 " method, and Q4_0 is a specific, popular implementation of that concept in llama.cpp.

Thank you.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment