Ziyan Gao

Ziyan Gao

Research Assistant Professor at Japan Advanced Institute of Science and Technology (JAIST).

Visual Prompt Guided Unified Pushing Policy

Published in arXiv preprint, 2026

Overview

This work proposes a unified pushing policy that incorporates a lightweight prompting mechanism into a flow matching policy to guide the generation of reactive, multimodal pushing actions. The visual prompt can be specified by a high-level planner, enabling the reuse of the pushing policy across a wide range of planning problems. Unlike existing approaches that rely on multi-step push plans composed of pre-defined pushing primitives with limited application scopes, this method provides greater efficiency and versatility.

Key Contributions

  • Unified Pushing Policy: A flexible framework that can adapt to different pushing scenarios through visual prompting
  • Lightweight Prompting Mechanism: Enables high-level planners to guide low-level pushing actions
  • Flow Matching Integration: Uses flow matching policies to generate reactive, multimodal pushing behaviors
  • VLM-guided Planning: Demonstrates integration with Vision Language Model-based planning for table-cleaning tasks

Model Architecture

Model Architecture

Figure 1: Policy Architecture. The input consists of the visual prompt and the latest T_obs steps of image data and robot proprioception. The policy is parameterized by a Diffusion Transformer with alternating self-attention and cross-attention DiT blocks to denoise action tokens A_0 into executable trajectories A_1.

Demonstration Video

For the demonstration video, please visit the Google Drive link to view the full demonstration of our Visual Prompt Guided Unified Pushing Policy.

Alternatively, you can watch the video directly if you have access to the shared file.

Publication Details

Authors: Hieu Bui, Ziyan Gao, Yuya Hosoda, Joo-Ho Lee

Co-Corresponding Authors: Hieu Bui & Ziyan Gao

Venue: arXiv preprint

Publication Date: 2026-02-24

arXiv: 2602.19193

*Equal contribution

Recommended citation: Bui, H.*, Gao, Z.*, Hosoda, Y., & Lee, J. (2026). Visual Prompt Guided Unified Pushing Policy. arXiv preprint arXiv:2602.19193. (* Equal contribution, co-corresponding authors)
Download Paper