Transformers documentation

SAM3

Transformers

You are viewing main version, which requires installation from source. If you'd like regular pip install, checkout the latest stable version (v5.22.1).

Join the Hugging Face community

and get access to the augmented documentation experience

Collaborate on models, datasets and Spaces

Faster examples with accelerated inference

Switch between documentation themes

to get started

该模型于 2025-11-19 贡献至 Hugging Face Transformers。

SAM3

概述

SAM3（Segment Anything Model 3）在 SAM 3: Segment Anything with Concepts 中提出。

SAM3 在图像上执行 可提示概念分割（PCS）。PCS 以文本和/或图像示例作为输入（例如，“黄色校车”），并预测与概念匹配的每个对象的实例掩码和语义掩码。

该论文的摘要如下：

我们提出了 Segment Anything Model (SAM) 3，这是一个统一的模型，能够基于概念提示（concept prompts）在图像和视频中对目标进行检测、分割和跟踪。我们将概念提示定义为短名词短语（例如，“yellow school bus”）、图像示例（image exemplars），或两者的组合。可提示概念分割（Promptable Concept Segmentation, PCS）接收此类提示，并返回所有匹配目标实例的分割掩码和唯一标识。为了推动 PCS 的发展，我们构建了一个可扩展的数据引擎，生成了一个包含 400 万个唯一概念标签的高质量数据集，涵盖图像和视频，并包含困难负样本。我们的模型由一个图像级检测器和一个基于记忆的视频跟踪器组成，二者共享同一个骨干网络。通过引入存在性头（presence head）将识别与定位解耦，从而提升了检测精度。SAM 3 在图像和视频 PCS 任务上的准确率是现有系统的两倍，并提升了先前 SAM 在视觉分割任务上的能力。我们开源了 SAM 3，以及用于可提示概念分割的新基准 Segment Anything with Concepts (SA-Co)。

该模型由 yonigozlan 和 ronghanghu 贡献。

使用 🤗 Transformers 的示例

仅文本提示

import requests
import torch
from PIL import Image

from transformers import Sam3Model, Sam3Processor


model = Sam3Model.from_pretrained("facebook/sam3", device_map="auto")
processor = Sam3Processor.from_pretrained("facebook/sam3")

# Load image
image_url = "http://images.cocodataset.org/val2017/000000077595.jpg"
image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")

# Segment using text prompt
inputs = processor(images=image, text="ear", return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model(**inputs)

# Post-process results
results = processor.post_process_instance_segmentation(
    outputs,
    threshold=0.5,
    mask_threshold=0.5,
    target_sizes=inputs.get("original_sizes").tolist()
)[0]

print(f"Found {len(results['masks'])} objects")
# Results contain:
# - masks: Binary masks resized to original image size
# - boxes: Bounding boxes in absolute pixel coordinates (xyxy format)
# - scores: Confidence scores

单个边界框提示

使用边界框对视觉概念中的对象进行分割：

# Box in xyxy format: [x1, y1, x2, y2] in pixel coordinates
# Example: laptop region
box_xyxy = [100, 150, 500, 450]
input_boxes = [[box_xyxy]]  # [batch, num_boxes, 4]
input_boxes_labels = [[1]]  # 1 = positive box

inputs = processor(
    images=image,
    input_boxes=input_boxes,
    input_boxes_labels=input_boxes_labels,
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    outputs = model(**inputs)

# Post-process results
results = processor.post_process_instance_segmentation(
    outputs,
    threshold=0.5,
    mask_threshold=0.5,
    target_sizes=inputs.get("original_sizes").tolist()
)[0]

多个框提示（正向和负向）

使用带有正负标签的多个框来细化概念：

# Load kitchen image
kitchen_url = "http://images.cocodataset.org/val2017/000000136466.jpg"
kitchen_image = Image.open(requests.get(kitchen_url, stream=True).raw).convert("RGB")

# Define two positive boxes (e.g., dial and button on oven)
# Boxes are in xyxy format [x1, y1, x2, y2] in pixel coordinates
box1_xyxy = [59, 144, 76, 163]  # Dial box
box2_xyxy = [87, 148, 104, 159]  # Button box
input_boxes = [[box1_xyxy, box2_xyxy]]
input_boxes_labels = [[1, 1]]  # Both positive

inputs = processor(
    images=kitchen_image,
    input_boxes=input_boxes,
    input_boxes_labels=input_boxes_labels,
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    outputs = model(**inputs)

# Post-process results
results = processor.post_process_instance_segmentation(
    outputs,
    threshold=0.5,
    mask_threshold=0.5,
    target_sizes=inputs.get("original_sizes").tolist()
)[0]

组合提示词（文本 + 负向框）

使用文本提示词配合负向视觉提示词来优化概念：

# Segment "handle" but exclude the oven handle using a negative box
text = "handle"
# Negative box covering oven handle area (xyxy): [40, 183, 318, 204]
oven_handle_box = [40, 183, 318, 204]
input_boxes = [[oven_handle_box]]

inputs = processor(
    images=kitchen_image,
    text=text,
    input_boxes=input_boxes,
    input_boxes_labels=[[0]],  # 0 = negative (exclude this region)
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    outputs = model(**inputs)

# Post-process results
results = processor.post_process_instance_segmentation(
    outputs,
    threshold=0.5,
    mask_threshold=0.5,
    target_sizes=inputs.get("original_sizes").tolist()
)[0]
# This will segment pot handles but exclude the oven handle

使用文本提示进行批量推理

高效处理具有不同文本提示的多张图像：

cat_url = "http://images.cocodataset.org/val2017/000000077595.jpg"
kitchen_url = "http://images.cocodataset.org/val2017/000000136466.jpg"
images = [
    Image.open(requests.get(cat_url, stream=True).raw).convert("RGB"),
    Image.open(requests.get(kitchen_url, stream=True).raw).convert("RGB")
]

# Different text prompt for each image
text_prompts = ["ear", "dial"]

inputs = processor(images=images, text=text_prompts, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model(**inputs)

# Post-process results for both images
results = processor.post_process_instance_segmentation(
    outputs,
    threshold=0.5,
    mask_threshold=0.5,
    target_sizes=inputs.get("original_sizes").tolist()
)

print(f"Image 1: {len(results[0]['masks'])} objects found")
print(f"Image 2: {len(results[1]['masks'])} objects found")

批量混合提示

在同一批次中为不同的图像使用不同的提示类型：

# Image 1: text prompt "laptop"
# Image 2: visual prompt (dial box)
box2_xyxy = [59, 144, 76, 163]

inputs = processor(
    images=images,
    text=["laptop", None],  # Only first image has text
    input_boxes=[None, [box2_xyxy]],  # Only second image has box
    input_boxes_labels=[None, [1]],  # Positive box for second image
    return_tensors="pt"
).to(model.device)

with torch.no_grad():
    outputs = model(**inputs)

# Post-process results for both images
results = processor.post_process_instance_segmentation(
    outputs,
    threshold=0.5,
    mask_threshold=0.5,
    target_sizes=inputs.get("original_sizes").tolist()
)
# Both images processed in single forward pass

语义分割输出

SAM3 还提供与实例掩码并行的语义分割：

inputs = processor(images=image, text="ear", return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model(**inputs)

# Instance segmentation masks
instance_masks = torch.sigmoid(outputs.pred_masks)  # [batch, num_queries, H, W]

# Semantic segmentation (single channel)
semantic_seg = outputs.semantic_seg  # [batch, 1, H, W]

print(f"Instance masks: {instance_masks.shape}")
print(f"Semantic segmentation: {semantic_seg.shape}")

在单张图像上进行高效的多提示词推理

在对同一图像运行多个文本提示时，请预先计算视觉嵌入，以避免冗余计算：

from transformers import Sam3Processor, Sam3Model
import torch
from PIL import Image
import requests


model = Sam3Model.from_pretrained("facebook/sam3", device_map="auto")
processor = Sam3Processor.from_pretrained("facebook/sam3")

# Load image
image_url = "http://images.cocodataset.org/val2017/000000077595.jpg"
image = Image.open(requests.get(image_url, stream=True).raw).convert("RGB")

# Pre-process image and compute vision embeddings once
img_inputs = processor(images=image, return_tensors="pt").to(model.device)
with torch.no_grad():
    vision_embeds = model.get_vision_features(pixel_values=img_inputs.pixel_values)

# Run multiple text prompts efficiently
text_prompts = ["ear", "eye", "nose"]
all_results = []

for prompt in text_prompts:
    text_inputs = processor(text=prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model(vision_embeds=vision_embeds, **text_inputs)
...
    results = processor.post_process_instance_segmentation(
        outputs,
        threshold=0.5,
        mask_threshold=0.5,
        target_sizes=img_inputs.get("original_sizes").tolist()
    )[0]
    all_results.append({"prompt": prompt, "results": results})

for item in all_results:
    print(f"Prompt '{item['prompt']}': {len(item['results']['masks'])} objects found")

在多个图像上进行高效的单提示推理

在对多张图片运行相同的文本提示时，请预先计算文本嵌入，以避免冗余计算：

from transformers import Sam3Processor, Sam3Model
import torch
from PIL import Image
import requests


model = Sam3Model.from_pretrained("facebook/sam3", device_map="auto")
processor = Sam3Processor.from_pretrained("facebook/sam3")

# Pre-compute text embeddings once
text_prompt = "ear"
text_inputs = processor(text=text_prompt, return_tensors="pt").to(model.device)
with torch.no_grad():
    text_embeds = model.get_text_features(**text_inputs)

# Load multiple images
image_urls = [
    "http://images.cocodataset.org/val2017/000000077595.jpg",
    "http://images.cocodataset.org/val2017/000000039769.jpg",
]
images = [Image.open(requests.get(url, stream=True).raw).convert("RGB") for url in image_urls]

# Run inference on each image reusing text embeddings
# Note: attention_mask must be passed along with text_embeds for proper masking
all_results = []

for image in images:
    img_inputs = processor(images=image, return_tensors="pt").to(model.device)
    with torch.no_grad():
        outputs = model(
            pixel_values=img_inputs.pixel_values,
            text_embeds=text_embeds,
            attention_mask=text_inputs.attention_mask,
        )
...
    results = processor.post_process_instance_segmentation(
        outputs,
        threshold=0.5,
        mask_threshold=0.5,
        target_sizes=img_inputs.get("original_sizes").tolist()
    )[0]
    all_results.append(results)

for i, results in enumerate(all_results):
    print(f"Image {i+1}: {len(results['masks'])} '{text_prompt}' objects found")

自定义分辨率推理

⚠️ **性能说明**：自定义分辨率可能会降低精度。该模型旨在以 1008px 分辨率使用。

如需更快的推理速度或更低的内存占用：

config = Sam3Config.from_pretrained("facebook/sam3")
config.image_size = 560
model = Sam3Model.from_pretrained("facebook/sam3", config=config, device_map="auto")
processor = Sam3Processor.from_pretrained("facebook/sam3", size={"height": 560, "width": 560})

提示标签约定

SAM3 使用以下标签约定：

对于点和框：

1：正向提示词（包含此区域/对象）
0：负面提示词（排除此区域/对象）
-10：批量输入的填充值

坐标格式：

输入框：[x1, y1, x2, y2]（xyxy 格式），以像素坐标表示
输出框（原始值）：[x1, y1, x2, y2]（xyxy 格式），归一化到 [0, 1]
输出框（后处理）：[x1, y1, x2, y2]（xyxy 格式），采用绝对像素坐标

Transformers

SAM3

概述

使用 🤗 Transformers 的示例

仅文本提示

单个边界框提示

多个框提示（正向和负向）

组合提示词（文本 + 负向框）

使用文本提示进行批量推理

批量混合提示

语义分割输出

在单张图像上进行高效的多提示词推理

在多个图像上进行高效的单提示推理

自定义分辨率推理

提示标签约定

Sam3Config

class transformers.Sam3Config

Sam3ViTConfig

class transformers.Sam3ViTConfig

Sam3VisionConfig

class transformers.Sam3VisionConfig

Sam3GeometryEncoderConfig

class transformers.Sam3GeometryEncoderConfig

Sam3DETREncoderConfig

class transformers.Sam3DETREncoderConfig

Sam3DETRDecoderConfig

类 transformers.Sam3DETRDecoderConfig

Sam3MaskDecoderConfig

class transformers.Sam3MaskDecoderConfig

Sam3Processor

class transformers.Sam3Processor

__call__

Sam3ImageProcessor

class transformers.Sam3ImageProcessor

预处理

Sam3ViTModel

class transformers.Sam3ViTModel

forward

Sam3VisionModel

class transformers.Sam3VisionModel

forward

Sam3Model

class transformers.Sam3Model

forward

get_text_features

call