Autognosi/sam3-onnx-vulkan

SAM 3 int8 ONNX

Five ONNX graphs provide two image workflows:

• Text prompts find and segment every matching object, such as all the plastic bottles in an image.
• Points and boxes produce a mask for one object selected with clicks or a bounding box, as in the original SAM workflow.

Video tracking is not included. These files process each image independently; they do not include SAM 3's video memory encoder or memory attention. The text detector also does not accept image exemplars or mask prompts.

The graphs were exported for onnx-vulkan-rs and use standard ONNX operators (opset 18). The included Python scripts run them with ONNX Runtime; the same graphs can run with other ONNX Runtime execution providers.

Run the examples

Install the Python dependencies, download this repository, then run a script against an image on your machine:

python -m pip install -U huggingface_hub onnxruntime numpy pillow tokenizers
uvx hf download Autognosi/sam3-onnx-vulkan --local-dir sam3-onnx-vulkan
cd sam3-onnx-vulkan

python segment.py photo.jpg "plastic bottle" --out overlay.png
python detect.py photo.jpg person dog > boxes.json
python classes.py --classes person dog bicycle -- *.jpg
python points.py photo.jpg --point 520,410 --out mask.png

photo.jpg and *.jpg refer to images in your current directory. Each script prints its options with --help. The command-line scripts apply EXIF rotation before inference. Coordinates use pixels in that displayed orientation. Python callers can use PIL.ImageOps.exif_transpose() for the same behavior.

Choose a workflow

Script	Input	Result
segment.py	One or more short text prompts	Masks, scores and boxes for matching objects; can save an overlay PNG
detect.py	One or more short text prompts	JSON boxes and scores; the script omits the masks
classes.py	A list of classes and one or more images	JSON detections per image; can save labelled overlays
points.py	Positive/negative clicks, a box, or both	One object mask, predicted IoU and object-presence result

Text prompts find each matching instance. A box selects one object inside the box. To find every instance of an object, use a text prompt.

For classes.py, pass class names with --classes or put one name per line in a file and pass it with --file. SAM 3 scores each class independently, so the script merges detections whose masks overlap by at least --overlap (default 0.5), keeping the higher-scoring one.

Use from Python

sam3_onnx.py loads only the graphs needed for each workflow. The default execution provider is ONNX Runtime CPU.

from PIL import Image
from sam3_onnx import Sam3

model = Sam3()
image = Image.open("photo.jpg").convert("RGB")

# Text path: encode the image once, then detect each prompt.
features = model.encode_image(image)
for detection in model.detect(features, "bottle", image.size):
    print(detection.score, detection.box, detection.mask.sum())

# Point/box path: encode once, then prompt for one object.
embeddings = model.embed_image(image)
mask, iou, present = model.segment(
    embeddings, image.size, points=[(520, 410)], negative=[(610, 300)]
)

segment() accepts image-pixel coordinates. Pass a box as box=(x0, y0, x1, y1). It returns a boolean mask, the decoder's predicted IoU and an object-presence flag. The same image embeddings can be reused for multiple objects.

Quality and performance

The quantized graphs were compared with the original PyTorch facebook/sam3 model in fp32. The text-path F1 uses box and mask IoU thresholds of 0.5; the point/box results report mask IoU. These measurements use four photos, and the photos were also used during development. They are a small evaluation set, not a substitute for testing on your own images.

Text-path F1 covers 16 image-prompt pairs and 44 reference detections. For the point/box path, prompts came from reference detections: a centre click, a click inside each detected object, and each detected box.

Workflow and runtime	Measurement
Text prompts, ONNX Runtime CPU, 16 image-prompt pairs	F1 0.934; 40 detections; mask IoU 0.981
Text prompts, ONNX Runtime CPU, 6 prompts on one photo	F1 0.985
Text prompts, onnx-vulkan-rs with f16 attention, 6 prompts on one photo	F1 0.976
Text prompts, PyTorch bf16, 6 prompts on one photo	F1 1.000
One click (best of three masks), ONNX Runtime CPU, 48 prompts	Mean mask IoU 0.961; minimum 0.435; 46/48 at IoU ≥ 0.9
Box, ONNX Runtime CPU, 44 prompts	Mean mask IoU 0.971; minimum 0.732; 40/44 at IoU ≥ 0.9

The text-path errors in this evaluation are concentrated in the cluttered prompts plastic, cable and floor.

On an RTX 4070, onnx-vulkan-rs took about 161 ms for the first text prompt on an image and 31 ms for each additional prompt, with all three text graphs resident in one process. PyTorch CUDA bf16 (eager) took 188.5 ms for the first prompt and 31.3 ms for each additional prompt on the same photo. This is a single-GPU measurement. The point and box graphs have not been timed.

Model files

The text workflow uses vision_encoder, text_encoder and detector. The point/box workflow uses tracker_vision_encoder and prompt_decoder. Each model.onnx stores its weights in the adjacent model.onnx.data file.

The two vision encoders share SAM 3's backbone but use different necks. Each workflow is self-contained, so the backbone is stored twice. To download only the point/box workflow and its Python support files:

uvx hf download Autognosi/sam3-onnx-vulkan --local-dir sam3-onnx-vulkan \
  --include "tracker_vision_encoder/*" "prompt_decoder/*" "*.py" LICENSE

Graph inputs, outputs and weight sizes

Graph	Inputs	Outputs	int8 weights (fp32 original)
vision_encoder/model.onnx	pixel_values f32 [1,3,1008,1008]	fpn_0 [1,256,288,288], fpn_1 [1,256,144,144], fpn_2 [1,256,72,72]	462 MB (1815 MB)
text_encoder/model.onnx	input_ids i64 [1,32], attention_mask i64 [1,32]	text_features f32 [1,32,256]	507 MB (1413 MB)
detector/model.onnx	fpn_0..2, text_features, attention_mask	pred_masks_bits u8 [1,200,288,36], pred_boxes [1,200,4], pred_logits [1,200], presence_logits [1,1], semantic_seg [1,1,288,288]	30 MB (102 MB)
tracker_vision_encoder/model.onnx	pixel_values f32 [1,3,1008,1008]	image_embeddings_0 [1,32,288,288], image_embeddings_1 [1,64,144,144], image_embeddings_2 [1,256,72,72]	483 MB (1815 MB)
prompt_decoder/model.onnx	image_embeddings_0..2, input_points f32 [1,P,N,2], input_labels i64 [1,P,N]	pred_masks [1,P,4,288,288], iou_scores [1,P,4], object_score_logits [1,P,1]	10 MB (22 MB)

tokenizer/ contains the tokenizer and processor configuration from facebook/sam3.

Quantization

Weights are quantized dynamically to int8 with one scale per output channel; activations are quantized at runtime. No calibration set was used. The text vision encoder quantizes its MatMul layers and neck convolutions, but not its patch embedding. The other four graphs quantize MatMul layers only. The text encoder keeps its token embedding table in fp32.

Each graph passed ONNX Runtime CPU correctness checks and the declared Vulkan tolerances. The maximum allowed quality drop against fp32 was 0.05 F1 for the text path and 0.05 mean mask IoU for the point/box path.

Graph	Gate metric	int8 / fp32	Evaluation samples
vision_encoder	Detection F1	0.982	4 images
text_encoder	Detection F1	0.961	9 prompts
detector	Detection F1	0.987	14 image-prompt pairs
tracker_vision_encoder	Mask IoU	0.974	4 images, 92 prompts
prompt_decoder	Mask IoU	0.987	92 prompts

quantization.json records export and quantization versions, options, gate results and SHA-256 hashes for the graph files. Quantizing every convolution in the text vision encoder reduced F1 by 0.056; quantizing the tracker vision encoder's convolutions reduced mask IoU by 0.046, compared with 0.026 for MatMul-only quantization.

Input processing and mask selection

Both workflows convert an image to RGB, resize it to 1008×1008 with bilinear interpolation, scale pixels to [0, 1], then normalize with mean 0.5 and standard deviation 0.5. The export uses this fixed input size.

Text workflow

• The tokenizer pads or truncates prompts to 32 tokens with <|endoftext|> (ID 49407).
• The score for each query is sigmoid(pred_logits) * sigmoid(presence_logits); the default threshold is 0.5.
• Boxes are normalized (x0, y0, x1, y1) coordinates and are returned in original-image pixels.
• The detector packs 200 masks at 288×288 into pred_masks_bits. The Python code unpacks them and resizes them to the original image size.

Point and box workflow

• Click coordinates use pixels in the displayed image. The script rescales them to the model's 1008×1008 input. The number of objects and points per object can vary between runs.
• Positive points have label 1; negative points have label 0. A box is encoded as its top-left and bottom-right corners with labels 2 and 3.
• The decoder returns four mask tokens. With exactly one positive click and no negative points or box, the code chooses the candidate with the highest predicted IoU. For other prompts it uses token 0 unless its stability ratio (logits > 0.05).sum() / (logits > -0.05).sum() is below 0.98; then it chooses the candidate with the highest predicted IoU. This follows the original model's mask-selection logic.
• The selected 288×288 mask logits are resized to the original image size and thresholded at zero. object present is true when object_score_logits > 0.

sam3_onnx.py implements this processing. Its tokenizer output matched the facebook/sam3 processor on the evaluation prompts; image preprocessing differed by at most one 8-bit level.

License and source

These files are derived from SAM 3 and are distributed under the SAM License, included in this repository. Use of SAM 3 materials must comply with that license, including section 1.b. Research using these files must acknowledge SAM 3 as required by the license.

Exported from facebook/sam3 at revision 3c879f39826c281e95690f02c7821c4de09afae7.