Key takeaways
- Zero-shot trajectory quality ranking exceeds the reported results of two specialized reward models. Without additional evaluation-task fine-tuning, GPT-6-Astra achieves a Kendall τa of 0.8166, above the reported values of 0.655 for Robometer and 0.675 for RynnValue-8B; direct preference accuracy reaches 92.4%.
- In-context examples substantially improve outcome judgments, but not every evaluation task benefits. Adding five examples raises RoboFail outcome accuracy from 64.91% to 85.96%; adding three reference trajectories for ranking lowers τa from 0.8166 to 0.7953.
- Progress scores can reflect task advancement and execution errors, but judgments of task success still contain mistakes. Zero-shot progress estimation reaches a VOC of 93.6. In the cases examined, scores fall after an object slips and rise after a successful retry, yet the model can also judge a successful endpoint as incomplete.
- Key-state recognition and failure diagnosis remain imperfect. The model can misread object states that determine success, miss the specific reason for failure, or assign equal scores to executions of different quality. These errors affect completion judgments, corrective feedback, and trajectory selection, respectively.
Introduction
Robot systems need to be able to evaluate their own behavior: assess how far a task has progressed, compare the quality of different execution trajectories, and identify and explain failures. General-purpose multimodal models can process visual observations alongside task descriptions, evaluation criteria, and examples. This offers a way to evaluate robot behavior without additional task-specific fine-tuning. In this report, we ask: To what extent can the existing visual understanding and reasoning capabilities of general-purpose multimodal models support the evaluation of robot behavior? How do these capabilities vary across evaluation tasks, available observations, and in-context examples?
We conduct an empirical study with GPT-6-Astra [1] as our primary subject on two benchmarks, RBM [2] [3] and RoboFail [4] . We examine trajectory quality scoring, direct preference judgments, task progress estimation, and subtask success and failure judgments with explanations, assessing the model’s ability to compare executions, track task progress, and recognize failures. We further compare zero-shot performance with conditions that provide in-context examples, and use cases with masked observations to examine how responses change with the available visual information.
Our results show that GPT-6-Astra is competitive at comparing execution quality, but still makes clear errors in completion judgment and failure diagnosis. Zero-shot ranking reaches Kendall τa = 0.8166, above the reported values of 0.655 for Robometer [2] and 0.675 for RynnValue-8B [5] ; direct preference accuracy is 92.4%. Adding five examples raises RoboFail outcome accuracy from 64.91% to 85.96%, while also increasing missed failures. In this article, we examine the state judgments, quality requirements, and error types behind these scores.
1. Study design
1.1 Evaluation tasks and benchmarks
We evaluate trajectory ranking, pairwise preference, and progress estimation on RBM [2] [3] , and subtask outcomes and failure explanations on RoboFail [4] .
1.2 Model configurations and evaluation conditions
Our primary subject is GPT-6-Astra [1] , supplied with task descriptions and observations through prompts, without additional evaluation-task fine-tuning. The RBM evaluation also includes zero-shot comparisons with GPT-5.6-sol [6] and DeepSeek-V4.1-Flash [7] .
Progress estimation, independent ranking, and RoboFail failure detection also include in-context learning (ICL) conditions; two RoboFail cases additionally compare responses under different numbers of visible frames.
2. Results
2.1 Trajectory quality and preference
We compare the three multimodal models on independent trajectory scoring and direct preference, then analyze GPT-6-Astra’s judgments through specific cases.
Videos in this section replay the eight model-input frames.
2.1.1 Trajectory quality ranking
We follow RBM-EVAL-OOD’s trajectory quality ranking protocol [2] [3] : score trajectories independently and use Kendall τa to measure agreement with the reference order “successful > suboptimal > failure” within each task. In the zero-shot setting, GPT-6-Astra, GPT-5.6-sol, and DeepSeek-V4.1-Flash achieve 0.8166, 0.7500, and 0.5300, respectively. All three general-purpose multimodal models distinguish execution quality, but their recognition capabilities differ substantially; GPT-6-Astra shows the strongest trajectory quality discrimination in this evaluation.
The ranking scores of GPT-6-Astra and GPT-5.6-sol also exceed the reported results for the specialized models RynnValue-8B (0.675) [5] and Robometer (0.655) [2] .
Case Study: distinguishing a closed drawer from one left open
In the xArm drawer-closing task, A is labeled successful and B is labeled failure. GPT-6-Astra assigns scores of 1.00 and 0.00, correctly distinguishing the quality of the two trajectories; DeepSeek assigns 0.20 and 0.30, reversing the reference order.
| Trajectory / reference label | GPT-6-Astra | DeepSeek-V4.1-Flash |
|---|---|---|
| A · success | 1.00 | 0.20 |
| B · failure | 0.00 | 0.30 |
In the input frames, A’s drawer front is pushed back and the drawer interior is no longer visible at the end; B remains clearly open. GPT-6-Astra identifies this final-state difference. DeepSeek instead claims that A’s drawer remains in its initial open position.
Read both models’ original reasons
- GPT-6-Astra · A · 1.00
- The robot pushes the protruding drawer inward until its front is flush with the cabinet frame and the open top is no longer visible. The final frames show the drawer closed.
- GPT-6-Astra · B · 0.00
- The drawer remains extended with its open interior clearly visible in the final frames. The robot gripper enters the drawer, but no sustained closing motion or closed final state is shown.
- DeepSeek-V4.1-Flash · A · 0.20
- The robot arm approaches and may contact the drawer handle, but the drawer remains in its initial open position; no visible closing motion or flush final state is achieved.
- DeepSeek-V4.1-Flash · B · 0.30
- Drawer moves slightly toward closed after initial handle contact, but remains visibly open in the final frame; no complete closure achieved.
Case Study: transferring the banana or the orange object
The MIT Franka task asks for the banana to be moved from the blue plate to the green plate. A is labeled successful and B suboptimal. GPT-6-Astra’s independent scores, A 1.00 > B 0.00, match that order; DeepSeek’s A 0.50 < B 1.00 reverse it.
| Trajectory / reference label | GPT-6-Astra | DeepSeek-V4.1-Flash |
|---|---|---|
| A · success | 1.00 | 0.50 |
| B · suboptimal | 0.00 | 1.00 |
A’s final frame shows the banana at the green plate’s rim. B moves the orange object to the green plate while leaving the banana on the blue plate. GPT-6-Astra distinguishes the target object; DeepSeek instead treats the object moved in B as the banana.
Read both models’ original reasons
- GPT-6-Astra · A · 1.00
- The robot grasps the banana from the blue plate, carries it to the green plate, and opens its gripper. The final frame shows the banana resting on the green plate near its rim, with the carrot left behind.
- GPT-6-Astra · B · 0.00
- The robot grasps and releases the orange object onto the green plate, not the banana. The yellow banana remains on the blue plate in the final frame, so the requested transfer is not completed.
- DeepSeek-V4.1-Flash · A · 0.50
- The robot correctly picks the banana (not the carrot) from the blue plate and moves it toward the green plate. However, the final frame shows the banana still held/at the green plate edge rather than clearly released and resting on the green plate; placement is incomplete.
- DeepSeek-V4.1-Flash · B · 1.00
- The robot grasps the banana from the blue plate, lifts it, and releases it onto the green plate; the final frame shows the banana resting on the green plate.
GPT-6-Astra’s ranking errors: inversions and ties
Case Study: moving the wrong object and tilting the plate
The task asks the robot to move the banana from the blue plate to the green plate. One execution moves the carrot, leaving the banana on the blue plate and the green plate empty; GPT-6-Astra gives it 0.00. Another lifts and tilts the blue plate, causing the banana to fall without reaching the green plate, yet receives 0.15. The model identifies that neither execution completes the goal, but the reference labels the former suboptimal and the latter failure, so the scores reverse the reference order.
| Trajectory | Reference quality | GPT-6-Astra score |
|---|---|---|
| Moving the carrot | Suboptimal | 0.00 |
| Banana falls after the plate is lifted | Failure | 0.15 |
The disagreement concerns the relative assessment of two incomplete behaviors, rather than whether the model notices that the goal is incomplete.
Read the original model responses
- Moving the carrot: 0.00
- The robot picks up the orange carrot rather than the yellow banana and releases it to the right of the green plate. The banana remains on the blue plate, and the green plate is empty.
- Banana slips after the plate is tilted: 0.15
- The robot approaches the banana but appears to lift and tilt the blue plate instead of grasping the banana. The banana slides or falls off to the right; no placement or release onto the green plate is visible.
Case Study: different quality, the same full score
For two trajectories that open the blue trash bin, the reference labels one successful and the other suboptimal, yet GPT-6-Astra gives both 1.00. For the successful execution, it cites the raised lid and clearly exposed interior; for the suboptimal one, it also treats the opening beneath the lid as evidence that the correct bin has been opened. The lost credit comes from a tie: the model recognizes the shared opening outcome but does not distinguish quality levels in its scores.
Read the original model responses
- Reference success: 1.00
- The left gripper lifts the blue bin's lid, leaving its dark interior visibly exposed in the final frames. The red and green bins remain closed.
- Reference suboptimal: 1.00
- The left gripper lifts the blue bin’s lid, leaving a visible opening beneath it in the final frame. The correct bin is opened; the red and green bins remain closed.
Effects of context examples: fewer inversions, more ties
Providing three reference trajectories at different score levels reduces inversions but increases ties: Kendall τa falls from 0.8166 to 0.7953, the inversion rate falls from 4.78% to 3.68%, and the tie rate rises from 8.78% to 13.11%.
Breakdown of ranking changes
Of 41 source–task groups, eight improve, ten decline, and 23 are unchanged. The figure decomposes concordant, tied, and inverted pairs using the scorer’s pair weights.
Correct order 86.44% · Tie 8.78% · Inversion 4.78%
Correct order 83.21% · Tie 13.11% · Inversion 3.68%
2.1.2 Direct preference
Direct preference presents two trajectories together and asks the model to choose the better execution. On 1,192 fixed trajectory pairs, GPT-6-Astra, GPT-5.6-sol, and DeepSeek-V4.1-Flash achieve accuracies of 92.4%, 90.7%, and 82.2%, respectively, averaged with equal weight across the six data sources. The gap between GPT-6-Astra and GPT-5.6-sol is 1.7 percentage points. As a reference, Robometer reports 82.5% accuracy [2] .
In the following case studies, GPT-6-Astra and DeepSeek receive the same task, images, and A/B order.
Case Study: button activation matters more than lingering contact
For this SO101 button-press pair, the actual input order is A: suboptimal; B: successful; reference preference: B. GPT-6-Astra selects B and DeepSeek selects A.
B’s input frames show the button lighting up after the gripper presses down, followed by withdrawal; A stays near the button without equally clear activation evidence. GPT-6-Astra chooses B based on activation. DeepSeek emphasizes A’s continued contact in the final frame and B’s withdrawal, leading to the opposite choice.
Read both models’ original reasons
- GPT-6-Astra · Actual choice B
- B lowers its gripper onto the button, which visibly lights up, then retracts. A approaches the button but shows no clear depression or activation.
- DeepSeek-V4.1-Flash · Actual choice A
- In A, the gripper descends onto the button and remains in contact with it at the final frame, indicating a press. In B, the gripper approaches the button but then retracts, leaving the button without clear sustained contact/pressing at the end.
Case Study: opening a lid is not the same as leaving it open
For this Koch red-bin pair, the actual input order is A: failed; B: successful; reference preference: B. GPT-6-Astra selects B, matching the reference; DeepSeek selects A, disagreeing with it.
Both inputs show a lid being lifted, but A opens in the penultimate sampled image and closes again in the last one; B’s final image still shows a raised lid and an opening into the bin. GPT-6-Astra explicitly identifies A’s lid falling shut. DeepSeek describes A as remaining open and lowers its assessment of B because the gripper remains near the opening.
Read both models’ original reasons
- GPT-6-Astra · Actual choice B
- Both executions lift the correct red bin's lid, but A's lid falls closed again in its final frame. B ends with the lid raised and the bin interior visibly exposed, achieving the requested open state.
- DeepSeek-V4.1-Flash · Actual choice A
- A's right arm lifts the red bin lid and retracts, leaving the red bin visibly open; B reaches the red bin but its gripper remains at/in the opening at the final frame, so the release/open state is less complete.
Case Study: unzipping the pencil case, but choosing the execution labeled failure
The task asks the robot to unzip the pencil case. GPT-6-Astra chooses the execution labeled failure: it says the slider moves toward the near end and leaves an opening behind it. For the successful execution, it instead describes the slider moving toward the far end while the zipper edges remain joined, as if the case were being zipped shut.
Read the original model responses
- The model chooses the reference failure (original response A)
- A moves the zipper slider toward the near end, leaving a visible opening behind it. B pulls the slider toward the far end and leaves the zipper edges joined, appearing to close rather than unzip the case.
2.1.3 Ambiguity in quality labels
Quality labels compress different dimensions of execution into three categories: successful, suboptimal, and failure. The following two preference disagreements concern the boundary between suboptimal and failure: whether manipulating the wrong object earns partial credit, and how to evaluate partial completion accompanied by spillage. GPT-6-Astra identifies these specific states, but the task descriptions and three-level labels do not specify the priorities or penalties attached to each criterion.
Case Study: moving the wrong object is “suboptimal”; failing to pick up the target is “failure”
The task asks the robot to move the banana from the blue plate to the green plate. A transfers the orange object to the green plate while leaving the banana on the blue plate, and is labeled suboptimal. B attempts grasps over the blue plate without picking up and transferring the banana, and is labeled failure. The reference therefore prefers A, whereas GPT-6-Astra chooses B, explicitly noting that neither completes the task and that B avoids transferring the wrong object.
The ambiguity is whether completing a transfer with the wrong object should earn more quality credit than failing to pick up the correct object. A completes grasping, transfer, and release when judged by the action sequence. If object correctness is a prerequisite for task progress, those actions do not advance the banana-transfer goal. A single “suboptimal > failure” order does not specify how to reconcile these standards.
Read the task and the verbatim model response
- Task instruction
- Pick up the banana from the blue plate and place it on the green plate
- GPT-6-Astra · Actual choice B · Reference preference A
- Neither execution completes the task: the banana remains on the blue plate in both. A grasps and releases the orange object onto the green plate instead; B makes grasp attempts without a confirmed pickup, but avoids transferring the wrong object.
Case Study: ranking a partial pour with spillage against no pour
The task asks the robot to pour the pebbles from the cup onto the plate. A tilts the cup and puts some pebbles on the plate, while spilling others onto the table and leaving some in the cup; it is labeled failure. B moves the cup to the plate but leaves it upright with pebbles inside; it is labeled suboptimal. The reference prefers B, whereas GPT-6-Astra chooses A, citing both the progress of pouring onto the plate and the spillage and remaining pebbles.
This comparison requires an explicit tradeoff between task progress and execution errors. How much credit should be given for pebbles reaching the target? How much should spillage and incomplete emptying subtract, and should those penalties place A below an execution that has not poured? The boundary between “suboptimal” and “failure” depends on these thresholds, which the three-level labels alone do not provide.
Read the task and the verbatim model response
- Task instruction
- Pour the pebbles from the cup onto the plate
- GPT-6-Astra · Actual choice A · Reference preference B
- A visibly tips the cup and leaves many pebbles on the plate, though several spill onto the table and some remain in the cup. B grasps and moves the cup onto the plate but leaves it upright with pebbles still inside, without a visible pour.
Interpretable quality levels require separate criteria for object correctness, goal completion, and collateral errors, together with explicit prerequisites and partial-credit rules. Flora-Rewarder provides an interface for scoring individual rubric steps [14] , which can be used to describe these requirements separately.
Sampling, metrics, and cited versions
Ranking and different-quality preference evaluation reuse Robometer’s official sampling and metric rules with seed=44. Of 571 cached trajectories, four belong to tasks with only successful examples and are excluded by the official rule, leaving 567. Preference sampling uses up to 500 pairs per source, yielding 1,192 pairs. Both metrics use equal weighting across the six sources.
This study uses seed=44; the current official default is 42. Both sample eight frames uniformly. GPT-6-Astra produces prompted scores or A/B choices, while Robometer uses trained progress/preference heads. We cite Robometer v1’s τa=0.655; the corresponding v2 value is approximately 0.643.
2.1.4 Summary
GPT-6-Astra’s zero-shot trajectory ranking exceeds the reported results of two specialized reward models, and direct preference accuracy reaches 92.4%.
In the error cases, inversions and ties account for most of the lost credit. Explicit criteria for task success and execution quality would help distinguish errors in state recognition from disagreements about the quality standard itself.
2.2 Task progress estimation
2.2.1 Zero-shot progress estimation
Beyond comparing whole executions, we examine whether general-purpose multimodal models can estimate task progress from the instruction and observations available so far. Across 105 successful trajectories, zero-shot VOC is 93.6, 93.5, and 89.7 for GPT-6-Astra, GPT-5.6-sol, and DeepSeek-V4.1-Flash, respectively.
GPT-6-Astra and GPT-5.6-sol have similar temporal correlations, both above DeepSeek; GPT-6-Astra is below Robometer’s reported 94.8 [2] .
The following replays show sampled frames, not original-speed video. Each prediction uses only observations available at that point; the hollow initial point is fixed at zero, and lines guide the eye.
Case Study: correctly distinguishing completion, slippage, and the wrong object
These positive cases show GPT-6-Astra recognizing task-relevant differences in state. For the same instruction to put the blue cup in the sink, it distinguishes successful placement, a slipping cup, and moving the wrong object, rather than treating all arm motion as progress.
GPT-6-Astra predicted progress
Replay frame 0 · Nearest sample 0 · Protocol initial value 0.00
Select a sample to seek; the hollow point is the fixed start.
GPT-6-Astra predicted progress
Replay frame 0 · Nearest sample 0 · Protocol initial value 0.00
Select a sample to seek; the hollow point is the fixed start.
GPT-6-Astra predicted progress
Replay frame 0 · Nearest sample 0 · Protocol initial value 0.00
Select a sample to seek; the hollow point is the fixed start.
In the successful execution, GPT-6-Astra gives 0.60 while the blue cup is still suspended outside the sink, then raises it to 1.00 after placement and release. In the slipping execution, the score is 0.55 near the sink and falls to 0.40 after the cup slips outside. When the wrong object is moved, the blue cup never moves and the final progress returns to 0.00. These rises and falls correspond to progress toward or setbacks from the task goal, rather than time passing alone.
Read the three original final-step responses
- Successful placement · 1.00
- The blue cup is now upright inside the sink basin, and the gripper has withdrawn above it. The instructed placement is complete.
- Slippage outside the sink · 0.40
- The cup has moved next to the sink but appears to have slipped from the gripper and is tilted outside the basin. It is closer to the destination, but must be grasped again and placed inside.
- Wrong object · 0.00
- The robot has moved the pale object into the sink, not the blue cup. The blue cup remains unmoved at its initial position on the counter.
Case Study: recognizing a setback, a retry, and completion
The task is to move the banana from the blue plate to the green plate, and this trajectory is labeled successful. After the first approach, the banana remains on the blue plate and the arm has withdrawn; GPT-6-Astra’s progress falls from 0.15 to 0.00. On the next approach, the score returns to 0.15. It reaches 1.00 only in the final sampled observation, when the banana is on the green plate and the gripper has moved away. The model neither mistakes a temporary setback for the final outcome nor declares completion solely because the robot approaches again.
GPT-6-Astra predicted progress
Replay frame 0 · Nearest sample 0 · Protocol initial value 0.00
Select a sample to seek; the hollow point is the fixed start.
Read the four original prediction rationales
- Stored sample 7 · Progress 0.15
- The gripper is positioned close above the banana, but the banana remains on the blue plate after the plate was disturbed. No secure pickup or transfer to the empty green plate is visible.
- Stored sample 15 · Progress 0.00
- The arm has withdrawn from the blue plate, losing the earlier close grasping position. The banana remains on the blue plate and the green plate is empty; no pickup or transfer has been achieved.
- Stored sample 23 · Progress 0.15
- The gripper has returned to the banana at the blue plate's edge, providing a close pickup position. The banana is not visibly lifted clear of the plate, and the green plate remains empty.
- Stored sample 31 · Progress 1.00
- The banana is now resting on the green plate, and the gripper has lifted away. The banana is no longer on the blue plate, completing the requested transfer.
2.2.2 Context examples and completion judgments
With one reference trajectory from the same or a similar task, annotated with stage scores, VOC rises from 93.6 to 94.2, a small increase in overall time correlation.
Case Study: higher correlation, worse completion judgment
In the zero-shot condition, the model judges the cap to be detached and gives final progress of 1.00; with one reference trajectory, it judges the cap to be still in place and gives final progress of 0.35. The latter condition has higher VOC, but also higher MSE.
| Condition | VOC ↑ | MSE ↓ | Final progress |
|---|---|---|---|
| Zero-shot | 90.51 | 0.0407 | 1.00 |
| Single-trajectory example | 95.36 | 0.1312 | 0.35 |
Read the two original endpoint judgments
- Zero-shot
- The bottle has tilted away from the white cap, exposing its threaded neck behind the cap. The cap is separated from the mouth, achieving the opening goal.
- Single-trajectory ICL
- The bottle remains braced between the foreground jaws while the upper gripper has shifted around it. The white cap still appears seated, with no clear loosening or removal to establish further progress.
Trajectory-level progress changes
Of 105 trajectories, VOC improves on 58, declines on 35, and is unchanged on 12; 11 of the 58 improvements have worse MSE. Aggregate time-reference MSE falls from 0.0402 to 0.0390. In banana transfer, VOC declines on 18 of 19 trajectories and is unchanged on one.
Complete result values and sources
| Configuration | Source | VOC ↑ | τa ↑ | Preference % ↑ |
|---|---|---|---|---|
| GPT-6-Astra (zero-shot) | This study | 93.6 | 0.8166 | 92.4 |
| GPT-6-Astra (ICL) | This study | 94.2 | 0.7953 | — |
| GPT-5.6-sol | This study | 93.5 | 0.7500 | 90.7 |
| DeepSeek-V4.1-Flash | This study | 89.7 | 0.5300 | 82.2 |
| RynnValue-8B | RynnValue Table 2 [5] | — | 0.6750 | — |
| RynnValue-4B | RynnValue Table 2 [5] | — | 0.6700 | — |
| Robometer / RBM-1M (paper v1) | Robometer Tables XIII XVI XVII [2] | 94.8 | 0.6550 | 82.5 |
| Robometer / RBM-1M (web) | Robometer website detailed OOD tables [3] | 94.0 | 0.6430 | 82.5 |
| Robometer / RoboReward data | Robometer paper [2] | 92.7 | 0.5520 | — |
| RoboReward-4B | Robometer paper [2] | 87.9 | 0.5020 | — |
| RoboReward-8B | Robometer paper [2] | 88.2 | 0.4650 | — |
| RoboDopamine-8B (web) | Robometer website [3] | 79.6 | 0.4530 | — |
| TOPReward / Qwen3-VL-8B | Robometer website [3] | 39.7 | 0.1280 | — |
| GVL / GPT-5-mini | Robometer paper [2] | 26.2 | 0.1990 | — |
| VLAC-8B | Robometer paper [2] | 37.5 | 0.0490 | — |
| VLAC-2B | Robometer paper [2] | — | 0.0770 | — |
| ReWiND / RBM-1M | Robometer paper [2] | 50.8 | 0.0140 | — |
| RL-VLM-F / GPT-5-mini | Robometer paper [2] | — | — | 55.5 |
The Robometer report [2] and website [3] give VOC of 94.8 / 94.0 and τa of 0.655 / 0.643, respectively. RBM-1M and RoboReward data identify training data; blank entries indicate unreported values.
2.2.3 Summary
GPT-6-Astra can track task-relevant progress and setbacks, but higher temporal correlation does not guarantee more accurate completion judgments.
VOC measures correlation between scores and normalized time. Judging task completion requires recognizing the specific state that constitutes completion, so the two should be evaluated separately.
2.3 Subtask success and failure
2.3.1 Overall performance and error distribution
RoboFail’s five-example ICL condition uses external examples selected and constructed manually. We select one subtask from each of five other real-robot trajectories in the original RoboFail dataset, forming two successful and three failed image–question–answer examples. The five examples and their order are fixed before these ICL requests; all 57 test questions use the same examples, without parameter fine-tuning. Outcome accuracy and composite ROUGE-L come from the same responses.
Sources, fixed order, and annotations of the five examples
The demonstrations come from the original real-robot RoboFail recordings released with REFLECT [4] . Our source and visual audit found that none of these five episodes appears among the 15 trajectory groups underlying the 57 test questions; no test images or answers serve as demonstrations. This correspondence is our audit, not an author-released split or identity mapping, and does not establish independence of environment, embodiment, or pretraining data.
| Fixed order | Source episode | Demonstrated subtask | Demonstration answer |
|---|---|---|---|
| 1 | appleInFridge1 | Pick up the bowl containing the apple | Failure: picked up the empty blue bowl |
| 2 | secureObjects1 | Pick up the knife | Success |
| 3 | appleInFridge2 | Put the bowl containing the apple inside the fridge | Failure: placed it on top of the fridge |
| 4 | appleInFridge4 | Pick up the apple | Success |
| 5 | heatPotato1 | Turn on the microwave | Failure: the door was still open |
The three failure answers adapt the authors’ episode-level failure descriptions and were checked against the original frames. The two success answers are our visual annotations of successful subtasks. Demonstration montages use raw RGB frames at author action boundaries, ordered in time with future regions left white. These are not AHA’s original five demonstrations, and exact image preprocessing is not guaranteed to match AHA.
The examples were fixed before this external five-example run and were not searched or replaced based on its ICL scores. Zero-shot results were already available during selection, so this was neither blinded nor preregistered sampling. Each request supplies the five image–question–answer pairs in order, followed by the current test question. All 57 test questions and their original inputs are retained.
We evaluate GPT-6-Astra on 57 RoboFail [4] subtask questions. Inputs are subtask descriptions and keyframe montages; outputs are Yes/No outcome judgments and textual explanations. ICL prepends a fixed set of five external image–question–reference-answer demonstrations to the test question; zero-shot prompting omits them.
Zero-shot accuracy is 64.91%, comparable to the reported results of the specialized AHA models. With ICL it reaches 85.96%, about 19.9 percentage points above the 66.1% reported for AHA-13B Vicuna [8] . The table below gives classification counts for both conditions.
| Reference → prediction | Zero-shot | Five-example ICL |
|---|---|---|
| Accuracy | 64.91% | 85.96% |
| Success → success | 5 | 21 |
| Success → failure (false alarm) | 18 | 2 |
| Failure → success (miss) | 2 | 6 |
| Failure → failure | 32 | 28 |
Full results and distinct RoboFail adaptations
The false-alarm rate falls from 78.26% to 8.70%, while the miss rate rises from 5.88% to 17.65% and failure recall falls from 94.12% to 82.35%.
Our source and visual audit found no overlap between the five demonstration episodes and test trajectories; see the demonstration provenance.
| Configuration | Source | Accuracy % ↑ | ROUGE-L ↑ |
|---|---|---|---|
| GPT-6-Astra / zero-shot | This study | 64.91 | 0.185 |
| GPT-6-Astra / ICL (5 examples) | This study | 85.96 | 0.499 |
| LLaVA-v1.5-13B | AHA Tables 3/4 [8] | 0.00 | 0.000 |
| LLaVA-NeXT-34B | AHA Tables 3/4 [8] | 1.70 | 0.018 |
| Qwen-VL | AHA Tables 3/4 [8] | 0.00 | 0.000 |
| Gemini-1.5 Flash | AHA Tables 3/4 [8] | 0.00 | 0.000 |
| GPT-4o | AHA Tables 3/4 [8] | 55.40 | 0.114 |
| GPT-4o / 5-shot | AHA Tables 3/4 [8] | 57.10 | 0.236 |
| AHA-7B | AHA Tables 3/4 [8] | 62.50 | 0.204 |
| AHA-13B / Llama-2 | AHA Tables 3/4 [8] | 64.30 | 0.280 |
| AHA-13B / Vicuna | AHA Tables 3/4 [8] | 66.10 | 0.290 |
The Vicuna and Llama-2 variants of AHA-13B are listed separately; published results use their respective evaluation samples.
Guardian [9] uses 153 execution and 30 planning examples; PRIMO [10] uses 100 Cosmos-curated examples. Their results are listed separately.
| Configuration | Execution accuracy % | Planning accuracy % |
|---|---|---|
| GPT-4o | 80 | 67 |
| GPT-4o + Sentinel | 80 | 63 |
| RoboFAC-7B | 25 | 5 |
| I-Fail-Sense-3B | 43 | 67 |
| Cosmos-Reason2-8B | 78 | 53 |
| CLIP+MLP + GF | 42 | 43 |
| I-Fail-Sense-3B + GF | 76 | 52 |
| Cosmos-Reason2-8B + GF | 82 | 70 |
| Guardian-8B + GF | 86 | 70 |
| Configuration | Accuracy % |
|---|---|
| Gemini 2.0 Flash | 67 |
| GPT-4o | 63 |
| OpenAI o1 | 61 |
| Claude-haiku-4.5 | 59 |
| Qwen2.5-VL-7B | 57.6 |
| Nemotron-H-56B | 64 |
| Cosmos-Reason1-7B | 60 |
| Cosmos-Reason1-56B | 66.2 |
| PRIMO (SFT) | 51 |
| PRIMO (RL) | 63 |
| PRIMO R1 | 67 |
2.3.2 Cases of outcome misclassification
Zero-shot detects 32 of 34 failures but labels 18 of 23 successes as failures. The cases below illustrate both types of error and their changes with ICL.
Videos show the original trajectories; the model receives the accompanying keyframe montage and the subtask question.
Case Study: Pressing the coffee-machine button
Zero-shot predicts failure because activation is not visible; the five-example ICL response matches the success label.
View input and model responses
Reference answer · Success
Yes, the robot succeed at the sub-task.
Zero-shot · False alarm
No. The robot arm is near the coffee machine, but it has not visibly pressed the button. The end effector appears misaligned with the control, and there is no visible indication that the machine has been activated.
ICL (5 examples) · Correct judgment
Yes, the robot successfully pressed the button on the coffee machine.
In the image, it contains many frames from a single view at different time steps in a matrix arrangement depicting the robot arm press the button on the coffee machine. Each frame is labeled with the corresponding timestep reflecting the temporal information from left to right. The white grid represents future sub-tasks that has yet to happen and should be ignore. For the given sub-tasks, first determine it has succeed by choosing from ["yes", "no"], and then explain the reason why the current sub-tasks has failed.
Case Study: Placing in a bowl: an unspecified target object
The reference attributes failure to the wrong object. Zero-shot instead cites an unreleased strawberry, while the five-example ICL response calls placement successful. The input question says only “place in the bowl,” without specifying the object.
View input and model responses
Reference answer · Failure
No, The robot execute the action on the wrong object.
Zero-shot · Correct judgment
No. The robot lowers the strawberry into the bowl but appears to still be holding it with the gripper. The placement is incomplete because the strawberry has not been released.
ICL (5 examples) · Missed failure
Yes, the robot successfully placed the strawberry in the bowl.
In the image, it contains many frames from a single view at different time steps in a matrix arrangement depicting the robot arm place in the bowl. Each frame is labeled with the corresponding timestep reflecting the temporal information from left to right. The white grid represents future sub-tasks that has yet to happen and should be ignore. For the given sub-tasks, first determine it has succeed by choosing from ["yes", "no"], and then explain the reason why the current sub-tasks has failed.
Additional contrast: both conditions recognize success
Case Study: Picking up an apple
Both conditions correctly recognize a successful grasp.
Reference answer · Success
Yes, the robot succeed at the sub-task.
Zero-shot · Correct judgment
Yes. The robot has successfully picked up the apple: in frame 2, the apple is held by the gripper and lifted clear of the countertop. No failure is visible.
ICL (5 examples) · Correct judgment
Yes, the robot successfully grasped the apple and lifted it off the countertop.
In the image, it contains many frames from a single view at different time steps in a matrix arrangement depicting the robot arm performing pick up of the apple. Each frame is labeled with the corresponding timestep reflecting the temporal information from left to right. The white grid represents future sub-tasks that has yet to happen and should be ignore. For the given sub-tasks, first determine it has succeed by choosing from ["yes", "no"], and then explain the reason why the current sub-tasks has failed.
2.3.3 In-context examples and visual observations
In the knob case, additional observations change the ICL judgment while the zero-shot judgment stays unchanged. The knob and slicing cases compare the first frame, first two frames, and all supplied frames, with one request per condition to examine response sensitivity to visual information.
Case Study: Grasping the stove knob
The reference identifies failure because the gripper does not close. Zero-shot predicts failure but cites grasp location; under the ICL condition, the model predicts success.
View input and model responses
Reference answer · Failure
No, The robot gripper fails to close the gripper.
Zero-shot · Correct judgment
No. The gripper is positioned over the burner area rather than around a stove knob, so it has not established a secure grasp on the knob.
ICL (5 examples) · Missed failure
Yes, the robot successfully grasped the stove knob.
In the image, it contains many frames from a single view at different time steps in a matrix arrangement depicting the robot arm grasp onto the stove knob. Each frame is labeled with the corresponding timestep reflecting the temporal information from left to right. The white grid represents future sub-tasks that has yet to happen and should be ignore. For the given sub-tasks, first determine it has succeed by choosing from ["yes", "no"], and then explain the reason why the current sub-tasks has failed.
With ICL, the response changes from failure with one or two frames to success with all four. Zero-shot predicts failure at all three observation levels.
View inputs and responses at each frame count
1 frames
Zero-shot
No. The gripper is positioned near the sink, away from the stove knobs, and has not closed around a knob.
ICL (5 examples)
No, the robot’s gripper is positioned away from the stove knobs and has not grasped one.
2 frames
Zero-shot
No. The gripper is positioned above the pot near the sink rather than around a stove knob, so it has not established a grasp on the target.
ICL (5 examples)
No, the robot grasped and lifted the pot instead of grasping the stove knob.
4 frames
Zero-shot
No. The robot moves its gripper toward the stove, but it does not securely grasp a stove knob. The gripper remains above the burner area rather than closing around the knob.
ICL (5 examples)
Yes, the robot successfully grasped the stove knob.
Case Study: Slicing a carrot
The reference labels success; both zero-shot and ICL conditions predict failure because no separated slice is visible.
View input and model responses
Reference answer · Success
Yes, the robot succeed at the sub-task.
Zero-shot · False alarm
No. The arm moves toward the carrot, but the blade does not complete a cut through it. The carrot remains intact on the board, with no visibly separated slice.
ICL (5 examples) · False alarm
No, the carrot remains whole on the counter; the robot did not successfully cut it into slices.
In the image, it contains many frames from a single view at different time steps in a matrix arrangement depicting the robot arm slice the carrot. Each frame is labeled with the corresponding timestep reflecting the temporal information from left to right. The white grid represents future sub-tasks that has yet to happen and should be ignore. For the given sub-tasks, first determine it has succeed by choosing from ["yes", "no"], and then explain the reason why the current sub-tasks has failed.
From one frame to all three, both conditions consistently predict failure, disagreeing with the success label.
View inputs and responses at each frame count
1 frames
Zero-shot
No. The carrot remains intact on the cutting board; the robot has not completed a cut or separated a slice.
ICL (5 examples)
No, the carrot remains whole on the counter; the robot has not cut it into slices.
2 frames
Zero-shot
No. The robot arm moves upward without visibly completing a cut through the carrot. The carrot remains on the cutting board, with no newly separated slice visible.
ICL (5 examples)
No, the robot picked up the knife but did not slice the carrot; the carrot remains whole on the countertop.
3 frames
Zero-shot
No. The arm moves toward the carrot, but the blade does not complete a cut through it. The carrot remains intact, with no visibly separated slice.
ICL (5 examples)
No, the robot placed the knife across the carrot but did not cut through it. The carrot remains whole.
2.3.4 Summary
With examples, fewer successful subtasks are labeled failures, but more failures are missed.
The gain in overall accuracy comes mainly from recognizing successful subtasks. If the evaluator is used to detect execution failures, failure detection must be assessed separately; overall accuracy alone does not establish how well it does this.
2.4 Failure explanations and scores
We analyze composite ROUGE-L for the same responses as Section 2.3, separating the contributions of success recognition and failure explanation.
ICL here uses exactly the same five manually constructed external demonstrations as outcome classification; examples were not reselected and the model was not queried again for the explanation metric.
2.4.1 Overall performance and baseline comparison
Composite ROUGE-L is 0.499 with five-example ICL versus 0.185 zero-shot. The success-recognition term rises from 0.0877 to 0.3684 (+0.2807), and the failure-explanation term from 0.0969 to 0.1305 (+0.0336).
GPT-6-Astra’s five-example ICL composite score is above the AHA-reported values for AHA-13B Vicuna (0.290), AHA-13B Llama-2 (0.280), AHA-7B (0.204), and five-shot GPT-4o (0.236) [8] .
Composite scoring rule and component calculations
Composite ROUGE-L does not measure failure explanations alone. Under the AHA rules [8] , a correct success answer receives 1; a correct failure receives stemmed ROUGE-L F1 against the reference explanation; and a label disagreement receives 0. Scores are summed and divided by 57. Evaluation uses the original English answers, not the Chinese paraphrases in this article.
The success-recognition term rises from 0.0877 to 0.3684. Its increase, 16 / 57 ≈ 0.2807, accounts for about 89.3% of the reported total gain of 0.3144. The failure-explanation term rises from 0.0969 to 0.1305, accounting for the remaining 10.7%. A score of 0.499 therefore does not mean that 49.9% of failure explanations are correct. Most of the gain comes from recognizing success.
2.4.2 Explanation scores and failure reasons
The failure-explanation component improves, but correctly detected failures decrease from 32 to 28.
Case Study: Picking up a knife: identifying the wrong object
Both conditions predict failure: zero-shot says the knife was not grasped, while ICL identifies a spoon, matching the reference’s wrong-object diagnosis.
Keyframe provenance and presentation. These two observations come from the existing input montage in the public AHA RoboFail test sample. For display, we crop the observation regions and arrange them chronologically, omitting white future placeholders. We neither resample the source video at uniform intervals nor ask a model to select new keyframes. The model receives the original montage and subtask question; the adjacent continuous video is provided for readers to review.
Pixel similarity and visual inspection match the two frames to approximately 30.00 / 37.63 seconds in the source video. These times locate the observations; they are not annotated failure intervals or model alarm times. The available public evidence does not establish whether the authors selected frames at fixed intervals, action boundaries, or by another rule, so fixed-interval sampling cannot be confirmed.
View input and model responses
Reference answer · Failure
No, The robot execute the action on the wrong object.
Zero-shot · Correct judgment
No. The knife remains on the countertop beside the sink while the robot arm lifts away without securing it. The grasp failed, so the knife was not picked up.
ICL (5 examples) · Correct judgment
No, The robot picked up the spoon instead of the knife.
In the image, it contains many frames from a single view at different time steps in a matrix arrangement depicting the robot arm performing pick up the knife. Each frame is labeled with the corresponding timestep reflecting the temporal information from left to right. The white grid represents future sub-tasks that has yet to happen and should be ignore. For the given sub-tasks, first determine it has succeed by choosing from ["yes", "no"], and then explain the reason why the current sub-tasks has failed.
In the knob case, zero-shot also detects failure, but emphasizes grasp location while the reference identifies an unclosed gripper.
2.4.3 Summary
Most of the composite ROUGE-L gain comes from success recognition, with a smaller increase in the failure-explanation term.
The knife and knob cases show that detecting failure does not guarantee a correct explanation: the explanation must identify the specific object, location, or action problem.
3. Discussion and outlook
3.1 Main findings
General-purpose models can competitively compare robot execution quality without additional evaluation-task fine-tuning. GPT-6-Astra’s zero-shot ranking and preference results are competitive with the cited specialized reward models, and GPT-5.6-sol’s ranking also exceeds both reported specialized-model values. Task descriptions and visual observations are enough to obtain competitive results on some execution-evaluation tasks, although results vary across general-purpose models.
In the cases examined, scores reflect task-relevant state changes, including mistakes and retries. Judgments in the drawer, button, and red-bin cases correspond to closure, activation, and a lid falling shut. Progress scores fall after the blue cup slips and rise after the banana transfer succeeds on a retry. These scores change with task events instead of simply increasing over time.
In the RoboFail experiment, five in-context examples substantially improve recognition of successful subtasks. Correct judgments on the 23 successful subtasks rise from 5 to 21. This improvement comes from adding examples to the input, without updating model parameters.
3.2 How and when GPT-6-Astra fails
The model can misread a key state and misjudge task completion. In the bottle-opening case, after receiving a reference trajectory, GPT-6-Astra describes the separated cap as still seated. The pencil-case and knob cases also show errors about whether the zipper is open or the gripper has closed. The model can describe the operation yet misread the state that determines completion; examples do not eliminate these errors.
The model can detect failure without identifying what went wrong. Zero-shot responses in the knife and knob cases correctly predict failure but miss the wrong object or unclosed gripper identified in the references. A correct outcome label alone is therefore insufficient for corrective feedback; the model also needs to identify the specific error accurately.
The model can recognize specific state changes, while the quality labels themselves remain ambiguous. In the banana-transfer and pebble-pouring cases, GPT-6-Astra distinguishes moving the wrong object from failing to pick up the target, and partial pouring with spillage from not pouring. Mapping these observations to “suboptimal” and “failure” requires a quality standard: does manipulating the wrong object earn partial credit, and how much should spillage offset progress toward the goal? Explicit priorities and thresholds would make the meaning of trajectory rankings more consistent.
3.3 Limitations and future work
This study evaluates models on only two benchmarks, RBM and RoboFail, with limited data and task coverage. The specialized baselines are typically designed specifically for the evaluation tasks covered by these benchmarks. Broad validation across datasets, tasks, and robot platforms is still lacking.
Reinforcement-learning validation. We will compare calibrated progress or outcome rewards with reward models trained from trajectory preferences. With policy initialization, algorithms, and interaction budgets fixed, we will compare against sparse environment rewards and specialized critics, reporting success rates, sample efficiency, and query costs across seeds. Independent environment criteria or human review will determine success, while checks for reward exploitation will examine apparent completion and other critic errors. These closed-loop experiments have not yet been conducted; this article reports no policy gains.
Evaluation settings and metrics
| Evaluation task | Inputs and outputs | Evaluation samples | Metrics |
|---|---|---|---|
| Trajectory quality rankingRBM-EVAL-OOD | InputTask description; eight sampled frames per trajectory OutputIndependent scores, ranked within each source–task group | 567 trajectories 41 source–task groups |
Kendall τa ↑ |
| Direct preference judgmentRBM | InputTwo trajectories for the same task; eight sampled frames each OutputChoice of the better trajectory | 1,192 trajectory pairs | Accuracy ↑Six sources, equally weighted |
| Task progress estimationRBM-EVAL-OOD | InputTask description; observations sampled up to the current time; previous scores OutputProgress score (0–1) and brief rationale | 105 successful trajectories Four predictions per trajectory |
VOC ↑ Time-reference MSE ↓ |
| Outcome judgment and explanationRoboFail | InputSubtask question; keyframe montage OutputOutcome label and failure explanation | 57 questions / 15 trajectory groups 23 successes / 34 failures |
Accuracy ↑ Failure recall ↑ False-alarm / miss rate ↓ Composite ROUGE-L ↑ |
RBM ranking protocol. We use RBM’s quality-label order and Kendall τa metric [2] , with 567 trajectories across 41 source–task groups in this run. The official sampler selects trajectories by task and quality label with configurable counts; 567 is this run’s sample count, not a fixed benchmark size.
RoboFail protocol. RoboFail was introduced by REFLECT [4] . We use the question–answer evaluation format provided by AHA [8] on the 57 questions recorded in this study. This set is not the entire original dataset and is not assumed to match other publications’ evaluation subsets exactly.
In-context examples. GPT-6-Astra receives task-specific context conditions:
- Progress estimation: one training trajectory of the same or a similar task, with six stage scores.
- Independent ranking: one randomly selected training trajectory at each of three score levels—high, middle, and low—without requiring a task match. The middle reference score is not the test set’s suboptimal label.
- RoboFail: five external image–question–answer demonstrations manually selected and constructed from five other original real-robot RoboFail episodes (two successes, three failures), not randomly sampled from a training split. Every test question uses the same examples and order; see the demonstration provenance.
Direct preference is evaluated only in the zero-shot setting.
Observation masking. Two RoboFail cases use the first frame, first two frames, or all observations, with one request per zero-shot and five-example ICL condition, for 12 requests. Partial observations have no time-specific labels; we compare responses without scoring accuracy.
Result sources. Chart tags [S], [P], [W], [R], and [A] denote this study, the Robometer paper, its project website, RynnValue, and AHA. RBM ranking and different-quality preference follow official sampling and metric rules; the details in Section 2.1 record seeds, model interfaces, and cited versions. RoboFail literature comparisons use different question sets, as described in Section 2.3.
Case comparisons. The four positive examples are selected post hoc for illustration, not to estimate overall error rates. GPT-6-Astra and DeepSeek receive identical tasks, trajectories, prompts, and input images without reference quality labels; preference cases retain the actual A/B order. Reasoning settings and compute budgets are unmatched: GPT-6-Astra uses low effort with a 512-token output limit, while DeepSeek uses max effort with a 32768-token limit. No swapped-order repeat or matched per-example Robometer output is available.
Replays and curves. Ranking and preference cases in Section 2.1 replay eight model-input images per trajectory at one image per second, using the last image as the poster. The added positive cases sample positions 0 / 4 / 9 / 13 / 18 / 22 / 27 / 31 from 32 stored frames. Progress cases in Section 2.2 replay 32 stored frames at four frames per second, with scores at positions 0 / 7 / 15 / 23 / 31. The initial zero is protocol-set; the other four points are predictions. These replays do not preserve original timing, and sample indices are not seconds. Lines guide the eye without adding predictions or human annotations between samples.
The progress protocol fixes the initial score at zero. Together with four model predictions, this gives five evaluation positions against the same time reference.
Independent ranking: Kendall τa. Within each source–task group, the reference order is success > suboptimal > failure. For pairs included by the official scorer, correct orders contribute +1, inversions −1, and ties 0, giving (C − D) / N. Scores are averaged across tasks and then equally across the six sources.
Direct preference: equal-source accuracy. Accuracy is computed within each source and then averaged equally across six sources. Here cₛ is the number of correct choices and nₛ the number of evaluated pairs in source s, with 1,192 pairs in total.
Progress: VOC and time-reference MSE. The RBM definition of VOC uses Pearson correlation. For one trajectory, let pᵢ be the prediction at evaluation position i and tᵢ the time reference from normalized frame position. Compute their correlation; the article reports it on a ×100 scale, higher being better. MSE measures numerical distance from the same time reference, lower being better.
VOC = 100 × r
MSE = (1 / m) Σᵢ(pᵢ − tᵢ)²
Here m is the number of evaluation positions; cross-source aggregation follows the official scorer. The reference is normalized frame position, not human-annotated semantic completion.
Outcome classification: accuracy, failure recall, false-alarm rate, and miss rate. Treat failure as positive: TP is a correctly detected failure, FN a missed failure, TN a correctly recognized success, and FP a success predicted as failure. Accuracy is (TP + TN) / 57; failure recall is TP / 34; false-alarm rate is FP / 23; and miss rate is FN / 34. All four are reported as percentages. They score the answers, not the robot’s execution success rate.
Failure explanations: AHA composite ROUGE-L, higher is better. A wrong label receives 0, a correct success receives 1, and a correct failure receives stemmed ROUGE-L F1 between the original English explanation and reference. For the latter, let L be the longest common subsequence length of the token sequences: P = L / generated-token count, R = L / reference-token count, and F1 = 2PR / (P + R), with zero for no overlap. Sum scores over 57 questions and divide by 57. The composite mixes success recognition and failure-text overlap. Section 2.4 separates them rather than treating the composite as explanation accuracy.
The following reference values follow from the metric definitions and label counts.
| Non-learning reference | Deducible result | Interpretation |
|---|---|---|
| Give every trajectory the same score | τa = 0 | No quality discrimination; predicted ties add no positive contribution. |
| Uniform random binary choice | Expected accuracy 50% | Applies only when exactly one of two choices is correct; not a measured run. |
| Always predict failure on RoboFail | 34 / 57 = 59.65% | Failure recall 100%, false-alarm rate 100%; recall alone is insufficient. |
| Output the time reference itself | VOC = 100; MSE = 0 | Theoretical values for a nonconstant reference; using the scoring target does not establish visual or completion understanding. |
References
- OpenAI. GPT-6 Astra. Model documentation.
- Liang et al. Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons. RSS, 2026.
- Liang et al. RBM-EVAL-OOD. 2026.
- Liu, Bahety, and Song. REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction. CoRL, 2023.
- Huang et al. RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance. 2026.
- OpenAI. GPT-5.6 Sol. Model documentation.
- DeepSeek. Introducing DeepSeek-V4.1-Flash: smarter, faster, more efficient. 2026.
- Duan et al. AHA: A Vision-Language-Model for Detecting and Reasoning over Failures in Robotic Manipulation. ICLR, 2025.
- Pacaud et al. Guardian: Detecting Robotic Planning and Execution Errors with Vision-Language Models. 2025.
- Liu et al. From Passive Observer to Active Critic: Reinforcement Learning Elicits Process Reasoning for Robotic Manipulation. ECCV, 2026.
- Zhang et al. An Unexpected Robot Policy: Early Evaluations of GPT-6 Astra on RoboDojo and Beyond. 2026.
- Yang, Khetan, and Zhang. Astranomy: An Empirical Study of GPT-6 Astra as a Native Robot Policy. 2026.
- Galbot Team. Systematically Exploring the Capabilities of GPT-6 Astra as Embodied Policies. 2026.
- NVIDIA. Flora-Rewarder. 2026.
- Zhai et al. A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learning. 2025.
- Lee et al. RoboReward: General-Purpose Vision-Language Reward Models for Robotics. 2026.
- Zhang et al. ReWiND: Language-Guided Rewards Teach Robot Policies without New Demonstrations. CoRL, 2025.
- Wang et al. RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback. ICML, 2024.
- Chen et al. TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics. 2026.
- Tan et al. Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulation. CVPR, 2026.
- Ma et al. Vision Language Models are In-Context Value Learners. ICLR, 2025.
Citation
To cite this article:
@misc{zhang2026beyondthescore,
title = {Beyond the Score: Early Evaluations of {GPT-6-Astra} as a Robot Critic},
author = {Zhang, Jinhao and Zhang, Yuwei and Zhou, Weitao and Tian, Honglong and Wang, Jiawei and Yu, Haibao and Liu, Xiaoyu and Ma, Jinming and Zuo, Yushen and Wei, Yuteng and Rui, Ke and Li, Minglei},
year = {2026},
month = oct,
organization = {Simple AI},
howpublished = {\url{https://cloud.simpleai.tech/simple-world-lab/beyond-the-score/}},
note = {Research blog}
}