Abstract
A diagnostic benchmark and two-phase reinforcement learning framework enable personalized vision-language assistants to jointly connect visual hazard evidence with personal memory for appropriate responses.
Personalized vision-language assistants use visual observations and personal memories to provide advice tailored to the user. When the observed scene contains a physical hazard, personal context, such as allergies, medical conditions, or mobility constraints, can change what help is appropriate. Responding appropriately therefore requires recognizing the visible hazard and applying the relevant person's information. We introduce PReMind-Bench, a diagnostic benchmark that evaluates this combination across diverse hazards and personal-memory conditions. It separately measures memory access, grounding, abstention, and hazard response, and assesses joint success on interdependency cases. Existing models often satisfy one requirement while failing the other, and training only for safety or personalization leaves a gap in joint success. We therefore propose PReMind, a two-phase reinforcement learning framework that combines on-demand and forced-call interactions (memory always retrieved) to jointly train grounded memory use and hazard response. At 4B, PReMind raises joint success from 12.9% to 28.6% over its base on the interdependency set and improves hazard adequacy and correct abstention on the main test. Joint-success gains persist under forced-call evaluation and under a second LLM judge across three Phase II seeds; averaged over these seeds, joint success also improves on unseen hazard groups. These findings highlight the need to evaluate and train personal assistants to connect visual evidence with the relevant person's context.
Community
We introduce PReMind, a framework for safety-aware visual personalization, featuring a diagnostic benchmark for jointly evaluating visual hazard response and grounded personal-memory use, and a two-phase RL post-training scheme that integrates adaptive memory access, entity grounding, and hazard handling.
Project page: https://oyt9306.github.io/premind.github.io/
Hazard plus personal memory is the right pairing to test: a photo of a nut-based snack means one thing for most people and something else for someone with an allergy. Showing that models often get one half right and miss the other is the useful part. For the abstention cases, did models more often over-refuse or answer without checking the memory?
Thanks for your interest in our work! We saw more of the latter—models often kept answering without the personal fact they needed. Some skipped the memory lookup; others looked but still answered when the relevant memory was missing. PReMind improved at asking for clarification or withholding unsupported personal advice, while also reducing unnecessary refusals from 7.9% to 1.6% compared with its base.
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- MemoProAgent: Toward Memory-Driven Proactive Assistance (2026)
- MemLife: Curating and Reasoning over Long-Term Egocentric Video Memories (2026)
- ReMAP: Restoring the Perceptual Cycle with Reasoning-Time Latent Visual Memory (2026)
- Generalized CycleGRPO for Fine-Grained Multimodal Understanding and Grounding (2026)
- Latent Multimodal Memory Reasoning (2026)
- Neos: Learning to Interleave Search and Visual Reasoning (2026)
- PerceptRL: Making Visual Perception Rewardable in MLLMs from Answers to Interactions (2026)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend
Get this paper in your agent:
hf papers read 2611.23808 Don't have the latest CLI?
curl -LsSf https://hf.co/cli/install.sh | bash Models citing this paper 1
Datasets citing this paper 1
Yeongtak/PReMind-Bench
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper