From Demonstrations to Symbolic Planning:
Robotic Scene Graph Generation via Symbolic-Consistent Temporal Abstraction

Accepted to IEEE-RAS International Conference on Humanoid Robots (Humanoids 2026)

Haoyi Zheng1, Daniel Sliwowski1, Dongheui Lee1,2
1 Autonomous Systems Lab, Vienna University of Technology (TU Wien)
2 Institute of Robotics and Mechatronics, German Aerospace Center (DLR)

Abstract

Robotic demonstrations provide a rich source of task knowledge, which could be leveraged for robotic task planning. Explicit symbolic representations offer a structured interface between robotic observations and symbolic planning, enabling task knowledge to be reused across different tasks. However, constructing planning-ready symbolic representations from robotic demonstrations remains challenging, particularly in achieving temporally consistent abstraction and accurately grounded symbolic states. We propose a framework that converts robotic demonstrations into planning-ready symbolic representations by first segmenting demonstrations into symbolically consistent segments and then generating a symbolic scene graph for each segment through robot-prior-informed reasoning. The resulting scene graphs are further lifted into reusable Planning Domain Definition Language (PDDL) operators for downstream symbolic planning. Experiments on a multi-task manipulation dataset show that our method produces more accurate and planning-ready scene graphs than existing zero-shot baselines while avoiding excessive VLM inference. We further demonstrate a pipeline that reuses the automatically lifted symbolic operators for planning on previously unseen real-world tasks.

Project Video

Overview of the proposed framework and experimental results.

Key Contributions

A symbolic boundary detection method

Segments robotic demonstrations into symbolically consistent segments accurately in a zero-shot manner.

A scene graph generation framework

Leverages manipulation object typing, robot observations, and physical support reasoning to generate planning-ready symbolic scene graphs.

A symbolic lifting pipeline

Induces reusable PDDL operators from grounded scene graphs, bridging foundation-model perception and symbolic task planning.

Framework Overview

From robot demonstrations to symbolic task planning and online generalization.

Overview of the proposed SG2SP framework
01

Symbolic Boundary Detection

Segment demonstrations into symbolic-consistent phases by detecting symbolic boundaries.

Section IV-A
02

Scene Graph Generation

Generate scene graphs through three stages: symbolic query generation, scene graph completion, and support relation completion.

Section IV-B
03

Symbolic Planner Induction

Convert the generated scene graphs into PDDL planning files for downstream symbolic task planning.

Section IV-C
04

Online Task Planning

Reuse the induced planning domain and scene graph generation framework to plan unseen tasks online.

Section IV-D

Scene Graph Visualizations

Explore scene graph visualization videos across different robots and manipulation tasks.

Induced PDDL Operators

Explore the induced symbolic operators, their preconditions, and their effects from the 12 tasks above.

PDDL ACTION SCHEMA

grasp_container

01 / 12

Parameters

Preconditions

Add Effects

Delete Effects

Prompts

Explore the prompts used in our framework.

01. Stage I: Prompt for Object Types Classification Scene Graph Generation
You are a robotics ontology assistant.
For each object, classify its manipulation properties.
Definitions:
- rigid:
	The object maintains its shape during normal 	manipulation, and does not have hinges or joints (e.g., 	cup, bottle, bowl, fruit, vegetable).
- articulated:
	The object contains movable joints or hinges (e.g., 	drawer, cabinet, door).
- fabric:
	The object is made of fabric and can be manipulated as 	such (e.g., cloth, towel, blanket).
- container:
	The object can contain another object (e.g., cup, bowl, 	bottle, drawer, cabinet, plate, box).
- movable:
	The object is normally movable by a tabletop robot.
        Objects:
        {object_list}

        Return ONLY valid JSON. Do NOT wrap the JSON inside markdown code fences.

        {{
            "object_name": {{
                "type": "rigid/fabric/articulated",
                "container": True/False,
                "movable": True/False
            }}
        }}
02. Stage II: Prompt for Scene Graph Completion Scene Graph Generation
You are a robotic scene analyst.
You are given:
        1. An RGB image of a tabletop manipulation scene.
        2. Detected movable objects with highlighted mask boundaries and labels.
        3. Current gripper states: {gripper_state_description}
        4. Current subtask: {current_action}. The image is taken during the middle of executing this subtask.

Your task is to complete the candidate symbolic scene graph, by selecting exactly ONE candidate for each relation and attribute from the provided lists.

Rules:
        - Consider holding to be true if the gripper is closed.
        - Inference should be consistent with the current subtask and gripper states.
        - Choose exactly ONE candidate for every relation and attribute.
        - If no relation holds, choose "none".
        - Do NOT invent new objects or predicates.

--------------------------------------------------
Complete the following symbolic graph template
--------------------------------------------------
{scene_graph_template}
        
Replace each candidate list with the selected value.
Return ONLY valid JSON.
03. Stage III: Prompt for Support Relation Completion Scene Graph Generation
You are a robotic scene analyst.
You are given:
    1. An RGB image of a tabletop manipulation scene.
    2. Detected objects with highlighted mask boundaries and labels.

The current symbolic scene graph is: {current_graph}
The following objects currently do not have a direct physical supporter: {missing_objects}

Task: Complete the scene graph by adding exactly ONE support relation for each listed object.

Rules:
    - Use only the predicates: {support-related predicates}.
    - Use only objects from the provided object list.
    - Do not modify existing relations.
    - Add only the missing support relations.
    - Do not return relations/attributes seen in the current symbolic scene graph.
    - Return ONLY valid JSON.

Object list: {objects_list}

    {{
        "relations":[
            ["object", "predicate", "supporter"]
        ]
    }}
04. Online Task Planning: Prompt for Planning Goal Generation Online Task Planning
You are an expert in PDDL planning.
You are given:
	1. A valid PDDL problem file with the goal section intentionally omitted.
	2. A human instruction describing the desired task.
Your job is to generate ONLY the PDDL goal section.
Rules:
	- Only use object names that already appear in the problem.
	- Only use predicates that are consistent with the problem/domain.
	- Do NOT invent new objects or predicates.
	- Do NOT modify any existing part of the problem.
	- Return ONLY content within the (:goal ...) block.
	- Do NOT include "(:goal" and ")".
	- Do NOT include explanations or markdown.
Human instruction: {user_instruction}
PDDL problem: {problem_without_goal}
Domain types: {domain_types}
Domain predicates: {domain_predicates}

Example output:
(and
    (predicate1 object_name1 object_name2)
    (not (predicate2 object_name3 object_name4))
)

Citation

BibTeX will be added when the final publication details are available.

Citation information forthcoming.