VLM task planner running...
System prompt
You are a helpful assistant for a legged robot equipped with a single arm and a two-finger gripper. You specialize in task planning, and you always suggest the plan that is most likely to fulfill the task. You learn and adapt from previous experience in the long-term memory and also the current short-term memory (especially the action failures). Always apply chain-of-thought reasoning by thinking step by step before making a final decision.
User prompt
The robot received this instruction from the user: Put the orange on the plate.
Here's the robot's current observation.
Here is the action history for the current task so far for reference:
{
"time_step": 1,
"action": {
"scene_description": "The scene shows a table with a white plate on the left and an orange partially obstructed by a fan on the right.",
"applicable_knowledge": "In a similar scenario, the robot learned to clear obstructions before attempting to pick up an object. The fan is obstructing the orange, so it should be moved first.",
"chain_of_thought_reasoning": "The task is to place the orange on the plate. The orange is partially obstructed by a fan, which could interfere with a pick action. From past experiences, clearing obstructions before picking up the target object is beneficial. Therefore, the first step should be to push the fan to the right to clear the area around the orange. This will allow a clear path for picking up the orange afterward.",
"chosen_action": "Push the fan to the right to clear the area around the orange.",
"chosen_skill": "push",
"target_object": "fan",
"push_direction": "right"
}
}
{
"time_step": 1,
"evaluation": {
"scene_description": "Before the action, the orange was partially obstructed by the fan, which was positioned to the right of the orange. The plate was on the left side of the table. After the action, the fan was moved further to the right, clearing the area around the orange, which remained in its original position. The plate is still on the left, unchanged.",
"is_action_successful": true,
"is_task_completed": false
}
}
{
"time_step": 2,
"action": {
"scene_description": "The scene shows a table with a white plate on the left and an orange in the center. The fan has been moved to the right, clearing the area around the orange.",
"applicable_knowledge": "In a similar scenario, the robot learned to clear obstructions before attempting to pick up an object. The fan has been moved, clearing the area around the orange, allowing a clear path for picking up the orange.",
"chain_of_thought_reasoning": "The task is to place the orange on the plate. The area around the orange is now clear, and there are no obstructions. The orange is not tiny or flat, so it can be picked up. The robot should pick up the orange and then place it on the plate. This sequence aligns with past successful experiences where objects were picked up and placed on another object.",
"chosen_action": "Pick up the orange.",
"chosen_skill": "pick",
"target_object": "orange",
"should_grasp_at_specific_section": false
}
}
{
"time_step": 2,
"evaluation": {
"scene_description": "Before the action, the orange was on the table, centered between a plate on the left and a fan on the right. After the action, the orange is held by the robot's gripper, lifted from its original position on the table.",
"is_action_successful": true,
"is_task_completed": false
}
}
This contains the actions proposed and executed, and the feedback after the action execution. Learn from the experience (including past and active logs) and adapt if needed.
Given the current scene and task, choose the next best action.
HARD CONSTRAINTS TO APPLY:
- PUSH only works on objects directly on the table (nothing can be between the object and the table). PICK and PLACE would also work.
- If the objective is to put an object on top of another object, you must use PICK and PLACE. PUSH would not work. Similarly, you cannot PUSH an object off another object because it is not directly on the table.
- When two objects are next to each other, it's infeasible to directly PUSH the object that is behind the other one.
- If the robot is holding something, it must PLACE that object before attempting to grasp another.
- When the target object is tiny or flat, which is hard to grasp, you cannot use PICK.
- In the case of action failure, never repeat the same action immediately without first rearranging the scene yourself. This is because the same failure is mostly likely to occur, whether the scene has been reset or not. Instead, consider whether other actions could be taken.
- Never give a suggestion on how to better execute a failed action—the robot cannot understand or adapt to such advice.
GENERAL RULES:
- Could interacting with other objects help? You are allowed to interact with any other objects on the table.
- Any object on the table can be used as a tool (all are clean and safe to handle). Be creative! But remember to always PICK up the tool first before using it.
- In the reasoning process, first propose a promising action, then check whether it violates any constraints, one by one. If it does, discard it and propose another. Repeat this until you find a promising action that satisfies all constraints.
ACTION PARAMETERS:
- For the pick up, if the object needs to be grasped at a specific section, you must specify that as well.
- For the place action, you need to specify which object to place the target object on (not next to). If the object needs to be placed at a specific section, you must specify that as well.
- For the push action, you need to specify the direction to push (left or right). When both push directions work, prefer pushing left if the object is on the left to the gripper; right if on the right.
Output your final decision in the specified structured format. A human operator may have reset the scene to its initial state after the failure.
Here are the past relevant experiences from long-term memory.
{
"scenario": "Instruction: Put the apple on the plate.\nScene: A table with a white plate on the left, a red apple in the center, and a cylindrical container on the right.",
"experience": "Experience: The robot was tasked with placing an apple on a plate. Initially, the robot's attempt to pick up the apple failed, likely due to interference from a nearby salt container. The robot then successfully pushed the salt container away, clearing the area around the apple. On the next attempt, the robot successfully picked up the apple and then placed it on the plate, completing the task. The lesson learned is to consider environmental factors and potential obstructions when a pick action fails, and to clear the area if necessary before retrying."
}
{
"scenario": "Instruction: Pick up the towel.\nScene: The towel was laid out flat on the table. There is also an orange on the table.",
"experience": "Experience: The robot was instructed to pick up the towel. First, it picked up the towel, which succeeded as the towel was grasped and lifted off the table."
}
{
"scenario": "Instruction: Pick up the plate, with or without the apple.\nScene: A pink plate with a red apple on it is centered on a table. The area around the plate is clear.",
"experience": "Experience: The robot was tasked with picking up a plate that had an apple on it. Initially, it tried to pick up both the plate and the apple together, which resulted in failure due to instability. The robot then learned to first pick up the apple and place it aside, which allowed it to successfully pick up the plate afterward."
}
{
"scenario": "Instruction: Move the meatskewer from the plate to the table.\nScene: The meatskewer is on the plate, and the table surface is clear.",
"experience": "Experience: The robot was instructed to move the meatskewer from the plate to the table. First, it picked up the meatskewer from the plate. Next, it placed it on the table. Finally, the action succeeded as the meatskewer was now centered on the table."
}
{
"scenario": "Instruction: Put the egg next to the banana.\nScene: An egg is on the table, centrally located, with a bunch of bananas to its right.",
"experience": "Experience: The robot was tasked with placing an egg next to a banana. Initially, it attempted to pick up the egg, which resulted in the egg breaking. Learning from this, the robot then successfully pushed the egg to the right, placing it next to the banana without breaking it."
}
They illustrate how the robot successfully planned action sequences to complete tasks, sometimes after initial failures. They reflect the robot’s capabilities and limitations. Always first carefully reason how the current scenario is similar to these past ones, including how the target object is similar. Identify any lessons learned that could apply to the current task. This is part of the chain-of-thought reasoning and helps you avoid repeating past failures.
Based on the above image, task and action, output your structured evaluation.
SUCCESS CRITERIA FOR ACTION AND TASK
- For picking up objects, if the object is not lying flat on the table at the original location, you might consider it a success. It can be tilted, still touching the table, or appear to be not lifted, and it still might be considered a success.
- For pushing an object toward another object, if these objects are getting much closer, you may consider it a success (they do not need to be touching each other or super close). If the push causes the target object to tip over (but not broken), it is still considered as a success.
- For placing, if the object is roughly placed on the target object, it might be considered as a success. It does not need to be perfectly placed on the center of the target object.
FAILURE CRITERIA FOR ACTION AND TASK
- Always a failure if the robot breaks or drops other objects unintentionally, you must consider it a failure for the current action and thus the overall task (regardless of how it performs).
- Action failure if the action clearly did not achieve its goal (e.g., gripper missed entirely, object didn’t move at all).
REASONING REQUIREMENTS (IF ACTION FAILED)
- Describe the scene before and after, especially spatial changes and object states.
- There is a success/failure for the action, and a success/failure for the overall task. Don't confuse them.
Here is the action history for the current task so far for reference:
Based on the above image, task and action, output your structured evaluation.
SUCCESS CRITERIA FOR ACTION AND TASK
- For picking up objects, if the object is not lying flat on the table at the original location, you might consider it a success. It can be tilted, still touching the table, or appear to be not lifted, and it still might be considered a success.
- For pushing an object toward another object, if these objects are getting much closer, you may consider it a success (they do not need to be touching each other or super close). If the push causes the target object to tip over (but not broken), it is still considered as a success.
- For placing, if the object is roughly placed on the target object, it might be considered as a success. It does not need to be perfectly placed on the center of the target object.
FAILURE CRITERIA FOR ACTION AND TASK
- Always a failure if the robot breaks or drops other objects unintentionally, you must consider it a failure for the current action and thus the overall task (regardless of how it performs).
- Action failure if the action clearly did not achieve its goal (e.g., gripper missed entirely, object didn’t move at all).
REASONING REQUIREMENTS (IF ACTION FAILED)
- Describe the scene before and after, especially spatial changes and object states.
- There is a success/failure for the action, and a success/failure for the overall task. Don't confuse them.
Based on the above image, task and action, output your structured evaluation.
SUCCESS CRITERIA FOR ACTION AND TASK
- For picking up objects, if the object is not lying flat on the table at the original location, you might consider it a success. It can be tilted, still touching the table, or appear to be not lifted, and it still might be considered a success.
- For pushing an object toward another object, if these objects are getting much closer, you may consider it a success (they do not need to be touching each other or super close). If the push causes the target object to tip over (but not broken), it is still considered as a success.
- For placing, if the object is roughly placed on the target object, it might be considered as a success. It does not need to be perfectly placed on the center of the target object.
FAILURE CRITERIA FOR ACTION AND TASK
- Always a failure if the robot breaks or drops other objects unintentionally, you must consider it a failure for the current action and thus the overall task (regardless of how it performs).
- Action failure if the action clearly did not achieve its goal (e.g., gripper missed entirely, object didn’t move at all).
REASONING REQUIREMENTS (IF ACTION FAILED)
- Describe the scene before and after, especially spatial changes and object states.
- There is a success/failure for the action, and a success/failure for the overall task. Don't confuse them.