Transformers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.22.2).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

SpQR

SpQR 量化算法采用了一种 16x16 分块的双层组 3 位量化结构，并包含稀疏离群值。

若要使用 SpQR 对模型进行量化，请参阅 Vahe1994/SpQR 仓库。

使用 from_pretrained() 加载 SpQR 量化模型。

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

quantized_model = AutoModelForCausalLM.from_pretrained(
    "elvircrn/Llama-2-7b-SPQR-3Bit-16x16-red_pajama-hf",
    dtype=torch.half,
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("elvircrn/Llama-2-7b-SPQR-3Bit-16x16-red_pajama-hf")

Update on GitHub

←Quark VPTQ→