Reinforcement learning relies on accurate reward functions, which are often hand-crafted or even unavailable in real-world applications, such as robotics. Recent work has explored the zero-shot reasoning capabilities of pre-trained Vision-Language Models (VLMs) as reward models. However, without careful prompt engineering, these approaches tend to produce suboptimal rewards, where false positive predictions can severely degrade downstream policy learning. In robotics, limited datasets comprising expert demonstrations are often collected to bootstrap policy learning. This scenario provides an opportunity to optimize a reward model prior to policy training. We propose Demo2Reward, a test-time adaptation technique to optimize the language instruction of a reward model based on a few demonstrations (3–10 trajectories) to reduce false positives while preserving true positives. Crucially, this requires no additional model training or computation resources during policy learning. We show that Demo2Reward consistently outperforms existing zero- and few-shot VLM reward models across a range of simulated robotic tasks and policy backbones. Finally, we demonstrate that Demo2Reward effectively transfers to a real-world robotic learning scenario, enabling policy learning without manually engineering a reward function.

A key failure mode of VLM reward models is false positive predictions: hallucinated rewards that falsely signal success. Such spurious rewards invite reward hacking: once a policy collects a false positive, it has no incentive to keep going and actually complete the task. Demo2Reward optimizes the prompt for a VLM success detector (VLM-SD), used as a zero-shot reward model in sparse-reward RL. During optimization, it primarily aims to suppress false positive predictions while granting a small bonus for every true positive it manages to preserve.

The resulting reward models are more cautious: they no longer overestimate the underlying binary rewards, predicting success only once the task is genuinely solved.

Improved reward functions translate directly into better policy performance. We verify this across simulation suites (Robomimic, MetaWorld) and a real robot, and across policy-learning backbones (IBRL, RLPD): training on Demo2Reward rewards consistently yields stronger policies than the baseline reward models.