IDEAS-Lab-Northwestern/pi05-base-stack-retrieve-joint-2cam-lora

π0.5 LoRA — Sim Stack-Retrieve (joint controller, 2-cam)

A LoRA fine-tune of π0.5 (pi05_base) for one ManiGuard simulated manipulation task, trained with the openpi trainer.

Task — stack-retrieve: pull the bottom object out of a same-object stack and move it into the green goal sphere. Prompts: flat object, chili pepper, bowl.

Model

• Warm-start: openpi pi05_base; LoRA on gemma_2b_lora + gemma_300m_lora. action_dim=32 (padded), action_horizon=16.
• Controller: JointController — 8-D joint state, 8-D absolute-joint action (7 arm joints trained as per-step deltas, gripper absolute; reconstructed to absolute at inference).
• Cameras (LIBERO 2-cam): base_0_rgb ← image_left (third-person overview) + left_wrist_0_rgb ← wrist; third image slot zero-filled + masked. Dataset is 3-cam rendered; image_right unused.

Data

sim-stack-retrieve-60-joint-3cam — 60 episodes / 48,208 frames; GELLO-teleop collected, re-rendered to joint + 3-cam.

Training

• 1x A100-80GB, bf16, full data-parallel (fsdp_devices=1).
• ~2 epochs: 4000 steps, batch 24, cosine LR (peak 4.3e-5, warmup 400, decay 4.3e-6).
• Final train loss = 0.01.

Checkpoints

Step folders 1000/ 2000/ 3000/ 4000/ — 4000 is the final ~2-epoch checkpoint.

Usage

Serve via the openpi policy server with config pi05_base_stack_retrieve_joint_2cam_lora; feed observation/image_left (overview) + observation/wrist_image + 8-D joint observation/state.