GAM-LIBERO-PRO

GAM (Grounded Action Model) grounds the objects named in the instruction with an open-vocabulary 3D detector (WildDet3D), turns each object into a 3D box, a point cloud and a handful of image tokens, and predicts action chunks with a flow-matching transformer. This repository holds everything the released code needs to run GAM on LIBERO / LIBERO-PRO: the LIBERO-tuned detector, one policy for all four suites, its span head and the precomputed training cache. All files are referenced from configs/libero_pro.yaml and download on first use; the LIBERO-PRO task files (bddl + initial states) are in gam-libero-pro-assets.

The videos are evaluation rollouts of the released policy on LIBERO-PRO cells (suite / perturbation); the wireframes are the 3D boxes the policy was given at that step, labelled with the object phrase they were grounded from (pick slot blue, place slot green, second pair orange / purple).

spatial / language	spatial / object	spatial / swap
object / object	object / swap	goal / object
goal / task	libero_10 / object	libero_10 / task

Quick Links

• 📄 Paper: Grounded Action Model: 3D Grounding as a Foundation for Robotics
• 💻 Code: GehaoZhang6/Grounded-Action-Model
• 📂 Models: GAM-RoboTwin, GAM-WildDet3D (general detector + universal span head)
• 📂 Task files: gam-libero-pro-assets

Intended Use

Evaluate the released policy on the 16 LIBERO-PRO cells (spatial / object / goal / 10 x language / object / task / swap) and the original LIBERO suites, or train your own policy on the released cache without running the perception stack. The configuration is depth-input: the detector takes the simulator depth map in its RGB-D backend, object geometry is its predicted 3D box, point clouds come from its own depth. Objects are grounded from the instruction on the first frame (text prompts from the span head or the released lookup table) and followed with CoTracker.

Install

git clone https://github.com/GehaoZhang6/Grounded-Action-Model && cd Grounding-Action-Model
conda create -n gam python=3.11 && conda activate gam
bash scripts/install.sh --libero-pro /path/to/LIBERO-PRO
export LIBERO_PRO_ROOT=/path/to/LIBERO-PRO

Evaluate

# one cell (videos like the ones above are written next to the results)
python benchmarks/libero_pro/eval.py --config configs/libero_pro.yaml --suite object --variant language
# all 160 (suite, variant, task) units on several GPUs, then the score table
python benchmarks/libero_pro/run_all.py --config configs/libero_pro.yaml --gpus 0,1,2,3
python benchmarks/libero_pro/score.py   --config configs/libero_pro.yaml
# your own checkpoint
python benchmarks/libero_pro/eval.py --config configs/libero_pro.yaml --suite object --variant language --ckpt runs/libero_pro/latest.pt

eval.chunking: tempens (temporal ensembling, the released setting) or rtc (real-time chunking). perception.text_slots: table (the released instruction -> object-phrase tables) or span_head (checkpoints/span_head_libero_pro.pt on any instruction).

Train

# on the released cache (set train.cache_dir: hf://gehaozhang/gam-libero-pro/cache/ in the yaml)
python -m GAM.train --config configs/libero_pro.yaml
# from the LIBERO demonstrations (LIBERO_DATASETS): replay in the simulator, precompute, train
python benchmarks/libero_pro/precompute.py --config configs/libero_pro.yaml --suite object   # x4 suites
python -m GAM.train --config configs/libero_pro.yaml
# end-to-end (frozen detector, perception inside the training loop; only the object tracks are precomputed)
python benchmarks/libero_pro/track.py --config configs/libero_pro.yaml --suite object        # x4 suites
python -m GAM.train --config configs/libero_pro.yaml --e2e

Training on the cache needs about 9 GB of GPU memory at batch 64; evaluation under 8 GB.

Files

path	what
checkpoints/wilddet3d_libero_pro.pt	WildDet3D fine-tuned on LIBERO + LIBERO-PRO scenes — 4.7 GB
checkpoints/gam_policy_libero_pro.pt	GAM policy for all four suites (step-6600 snapshot, EMA weights) — 1.2 GB
checkpoints/span_head_libero_pro.pt	T5 span head: instruction -> object phrases, fine-tuned on LIBERO instructions
cache/libero_<suite>__<task>/demo_<i>.npz	precomputed training cache: 4 suites x 10 tasks x 50 demonstrations

Cache format

cache/libero_<suite>__<task>/demo_<i>.npz: state (T,8), action (T,7), patches (T,36,1024) float16 (a 6x6 cell is kept if a target 2D box covers at least 30 % of it or the robot silhouette is in it), box2d (T,4,4), xyz (T,4,3), boxgeo (T,4,9) (extents + rot6d), pcd_pts (T,4,512,3) float16, pcd_mask (T,4,512), clicks (T,4,2), kept (demo frame indices after still-frame removal), instruction, slot_names, slot_kinds. The detector's patch tokens come from an RGB-D encoder, so the training cache and the evaluation must use the same detector_depth.

License

The GAM code is MIT (GehaoZhang6/Grounded-Action-Model). The WildDet3D weights here are fine-tuned from WildDet3D (Allen Institute for AI), which builds on SAM 3 and LingBot-Depth, and are distributed under the SAM License; the policy checkpoints, span heads and caches are MIT.

Citation

@misc{zhang2026groundedactionmodel3d,
  title         = {Grounded Action Model: 3D Grounding as a Foundation for Robotics},
  author        = {Gehao Zhang and Weikai Huang and Shailesh Shailesh and Yiyan Peng and Jiafei Duan and Ranjay Krishna},
  year          = {2026},
  eprint        = {2609.23863},
  archivePrefix = {arXiv},
  primaryClass  = {cs.RO},
  url           = {https://arxiv.org/abs/2609.23863}
}