Transformers documentation

Qwen2-VL

Transformers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.22.1).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

Model ini dipublikasikan di HF papers pada 2024-09-18 dan disumbangkan ke Hugging Face Transformers pada 2024-08-26.

Qwen2-VL

Ikhtisar

Model Qwen2-VL (blog post) adalah pembaruan besar dari Qwen-VL yang dikembangkan oleh tim Qwen di Alibaba Research.

Abstrak dari blog tersebut adalah sebagai berikut:

Blog ini memperkenalkan Qwen2-VL, versi lanjutan dari model Qwen-VL yang telah mengalami peningkatan signifikan selama satu tahun terakhir. Peningkatan utama meliputi pemahaman gambar yang lebih baik, pemahaman video yang lebih canggih, fungsi agen visual terintegrasi, serta dukungan multibahasa yang diperluas. Arsitektur model telah dioptimalkan untuk menangani resolusi gambar apa pun melalui dukungan Naive Dynamic Resolution dan memanfaatkan Multimodal Rotary Position Embedding (M-ROPE) untuk memproses data teks 1D dan data visual multidimensi secara efektif. Model yang diperbarui ini menunjukkan kinerja yang kompetitif dibandingkan sistem AI terkemuka seperti GPT-4o dan Claude 3.5 Sonnet dalam tugas-tugas terkait visi, serta menempati peringkat tinggi di antara model open-source dalam kemampuan teks. Kemajuan ini menjadikan Qwen2-VL sebagai alat serbaguna untuk berbagai aplikasi yang membutuhkan kemampuan pemrosesan multimodal dan penalaran yang kuat.

Qwen2-VL architecture. Taken from the blog post.

Model ini disumbangkan oleh simonJJJ.

Contoh penggunaan

Inferensi Media Tunggal

Model dapat menerima gambar dan video sebagai input. Berikut adalah contoh kode untuk inferensi.

from transformers import AutoProcessor, Qwen2VLForConditionalGeneration


# Load the model in half-precision on the available device(s)
model = Qwen2VLForConditionalGeneration.from_pretrained("Qwen/Qwen2-VL-7B-Instruct", device_map="auto")
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct")


conversation = [
    {
        "role":"user",
        "content":[
            {
                "type":"image",
                "url": "https://huggingface.co/datasets/hf-internal-testing/transformers-synthetic-assets/resolve/main/images/qwen_vl_demo.jpeg"
            },
            {
                "type":"text",
                "text":"Describe this image."
            }
        ]
    }
]

inputs = processor.apply_chat_template(
    conversation,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device)

# Inference: Generation of the output
output_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
print(output_text)



# Video
conversation = [
    {
        "role": "user",
        "content": [
            {"type": "video", "path": "/path/to/video.mp4"},
            {"type": "text", "text": "What happened in the video?"},
        ],
    }
]

inputs = processor.apply_chat_template(
    conversation,
    fps=1,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device)


# Inference: Generation of the output
output_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
print(output_text)

Inferensi Campuran Media Batch

Model dapat memproses input dalam batch yang terdiri dari sampel campuran berbagai jenis, seperti gambar, video, dan teks. Berikut adalah contohnya.


# Conversation for the first image
conversation1 = [
    {
        "role": "user",
        "content": [
            {"type": "image", "path": "/path/to/image1.jpg"},
            {"type": "text", "text": "Describe this image."}
        ]
    }
]

# Conversation with two images
conversation2 = [
    {
        "role": "user",
        "content": [
            {"type": "image", "path": "/path/to/image2.jpg"},
            {"type": "image", "path": "/path/to/image3.jpg"},
            {"type": "text", "text": "What is written in the pictures?"}
        ]
    }
]

# Conversation with pure text
conversation3 = [
    {
        "role": "user",
        "content": "who are you?"
    }
]


# Conversation with mixed media
conversation4 = [
    {
        "role": "user",
        "content": [
            {"type": "image", "path": "/path/to/image3.jpg"},
            {"type": "image", "path": "/path/to/image4.jpg"},
            {"type": "video", "path": "/path/to/video.jpg"},
            {"type": "text", "text": "What are the common elements in these media?"},
        ],
    }
]

conversations = [conversation1, conversation2, conversation3, conversation4]
# Preparation for batch inference
inputs = processor.apply_chat_template(
    conversations,
    fps=1,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt"
).to(model.device)


# Batch Inference
output_ids = model.generate(**inputs, max_new_tokens=128)
generated_ids = [output_ids[len(input_ids):] for input_ids, output_ids in zip(inputs.input_ids, output_ids)]
output_text = processor.batch_decode(generated_ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
print(output_text)

Tips Penggunaan

Kompromi Resolusi Gambar

Model ini mendukung berbagai rentang resolusi input. Secara default, model menggunakan resolusi native untuk input, namun resolusi yang lebih tinggi dapat meningkatkan performa dengan biaya komputasi yang lebih besar. Pengguna dapat mengatur jumlah piksel minimum dan maksimum untuk mencapai konfigurasi optimal sesuai kebutuhan mereka.

min_pixels = 224*224
max_pixels = 2048*2048
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct", min_pixels=min_pixels, max_pixels=max_pixels)

Jika RAM GPU terbatas, resolusi dapat dikurangi sebagai berikut:

min_pixels = 256*28*28
max_pixels = 1024*28*28
processor = AutoProcessor.from_pretrained("Qwen/Qwen2-VL-7B-Instruct", min_pixels=min_pixels, max_pixels=max_pixels)

Hal ini memastikan setiap gambar dikodekan menggunakan jumlah token antara 256-1024. Angka 28 berasal dari fakta bahwa model menggunakan ukuran patch 14 dan ukuran patch temporal 2 (14 x 2 = 28).

Input Gambar Berganda

Secara default, konten gambar dan video disertakan langsung dalam percakapan. Saat menangani beberapa gambar, sangat membantu untuk menambahkan label pada gambar dan video agar lebih mudah dirujuk. Pengguna dapat mengontrol perilaku ini dengan pengaturan berikut:

conversation = [
    {
        "role": "user",
        "content": [
            {"type": "image"},
            {"type": "text", "text": "Hello, how are you?"}
        ]
    },
    {
        "role": "assistant",
        "content": "I'm doing well, thank you for asking. How can I assist you today?"
    },
    {
        "role": "user",
        "content": [
            {"type": "text", "text": "Can you describe these images and video?"},
            {"type": "image"},
            {"type": "image"},
            {"type": "video"},
            {"type": "text", "text": "These are from my vacation."}
        ]
    },
    {
        "role": "assistant",
        "content": "I'd be happy to describe the images and video for you. Could you please provide more context about your vacation?"
    },
    {
        "role": "user",
        "content": "It was a trip to the mountains. Can you see the details in the images and video?"
    }
]

# default:
prompt_without_id = processor.apply_chat_template(conversation, add_generation_prompt=True)
# Excepted output: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>Hello, how are you?<|im_end|>\n<|im_start|>assistant\nI'm doing well, thank you for asking. How can I assist you today?<|im_end|>\n<|im_start|>user\nCan you describe these images and video?<|vision_start|><|image_pad|><|vision_end|><|vision_start|><|image_pad|><|vision_end|><|vision_start|><|video_pad|><|vision_end|>These are from my vacation.<|im_end|>\n<|im_start|>assistant\nI'd be happy to describe the images and video for you. Could you please provide more context about your vacation?<|im_end|>\n<|im_start|>user\nIt was a trip to the mountains. Can you see the details in the images and video?<|im_end|>\n<|im_start|>assistant\n'


# add ids
prompt_with_id = processor.apply_chat_template(conversation, add_generation_prompt=True, add_vision_id=True)
# Excepted output: '<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\nPicture 1: <|vision_start|><|image_pad|><|vision_end|>Hello, how are you?<|im_end|>\n<|im_start|>assistant\nI'm doing well, thank you for asking. How can I assist you today?<|im_end|>\n<|im_start|>user\nCan you describe these images and video?Picture 2: <|vision_start|><|image_pad|><|vision_end|>Picture 3: <|vision_start|><|image_pad|><|vision_end|>Video 1: <|vision_start|><|video_pad|><|vision_end|>These are from my vacation.<|im_end|>\n<|im_start|>assistant\nI'd be happy to describe the images and video for you. Could you please provide more context about your vacation?<|im_end|>\n<|im_start|>user\nIt was a trip to the mountains. Can you see the details in the images and video?<|im_end|>\n<|im_start|>assistant\n'

Flash-Attention 2 untuk mempercepat proses generasi

Pertama, pastikan untuk menginstal versi terbaru dari Flash Attention 2:

pip install -U flash-attn --no-build-isolation

Selain itu, Anda harus memiliki perangkat keras yang kompatibel dengan Flash-Attention 2. Baca lebih lanjut tentang hal ini di dokumentasi resmi flash attention repository. FlashAttention-2 hanya dapat digunakan ketika model dimuat dalam torch.float16 atau torch.bfloat16.

Untuk memuat dan menjalankan model menggunakan Flash Attention-2, cukup tambahkan attn_implementation="flash_attention_2" saat memuat model sebagai berikut:

from transformers import Qwen2VLForConditionalGeneration

model = Qwen2VLForConditionalGeneration.from_pretrained(
    "Qwen/Qwen2-VL-7B-Instruct", 
    attn_implementation="flash_attention_2",
 device_map="auto")

Transformers

Qwen2-VL

Ikhtisar

Contoh penggunaan

Inferensi Media Tunggal

Inferensi Campuran Media Batch

Tips Penggunaan

Kompromi Resolusi Gambar

Input Gambar Berganda

Flash-Attention 2 untuk mempercepat proses generasi

Qwen2VLConfig

kelas transformers.Qwen2VLConfig

Qwen2VLVisionConfig

kelas transformers.Qwen2VLVisionConfig

Qwen2VLTextConfig

kelas transformers.Qwen2VLTextConfig

Qwen2VLImageProcessor

kelas transformers.Qwen2VLImageProcessor

praproses

Qwen2VLVideoProcessor

kelas transformers.Qwen2VLVideoProcessor

praproses

Qwen2VLImageProcessorPil

kelas transformers.Qwen2VLImageProcessorPil

praproses

Qwen2VLProcessor

kelas transformers.Qwen2VLProcessor

__call__

Qwen2VLTextModel

kelas transformers.Qwen2VLTextModel

forward

Qwen2VLModel

kelas transformers.Qwen2VLModel

forward

get_video_features

get_image_features

Qwen2VLForConditionalGeneration

kelas transformers.Qwen2VLForConditionalGeneration

forward

get_video_features

get_image_features

call