[RuntimeError] GPU is required to quantize or run quantize model – Qwen1.5-0.5B-Chat in my Space

John6666
Fri, 23 May 2025 15:57:10 +0000

It may be possible to use a quantized model in a CPU environment, but it would probably be faster to simply use a non-quantized model in this case.

#MODEL_ID = "Qwen/Qwen1.5-0.5B-Chat-GPTQ-Int4"
MODEL_ID = "Qwen/Qwen1.5-0.5B-Chat"

huggingface.co

GPTQ

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Loading quantized model on CPU only 🤗Transformers

Im currently trying to run BloomZ 7b1 on a server with ~31GB available ram. Without quantization loading the model starts filling up swap, which is far from desirable. I tried enabling quantization with load_in_8bit: from transformers import AutoTokenizer, AutoModelForCausalLM, TextStreamer import torch modelPath = "/mnt/backup1/BLOOM/" device = torch.device("cpu") tokenizer = AutoTokenizer.from_pretrained(modelPath) model = AutoModelForCausalLM.from_pretrained(modelPath, device_map="auto",…