HUMXAI · TECHNICAL REPORT · PHASE 1
Recovering a frozen VLA without retraining
A memory layer that retargets recovered demonstrations to the live object's frame rescues 46.7% of the episodes where SmolVLA fails under scene perturbation — without touching the base policy's weights, without per-task RL, and without the magnitude cliff that sinks replay-based corrections.
Base → Recovery
4.4% → 46.7%
21 of 45 rescued
McNemar exact p = 0.0001
Far field, 6–12 cm
0% → 55.0%
11 of 20 rescued, 0 broken
p = 0.0010
Trained parameters
0
Base policy frozen
analytic residual, no RL
Recovery latency
110 steps
median; range 92–126
from detector trigger
The problem
A vision-language-action policy learns to solve a task from the scenes it saw during training. Move the object a few centimeters from where it sat in the demonstrations, and the policy doesn't degrade gracefully — it breaks. On our test bank, a frozen SmolVLA solves the task in 2 of 45 perturbed scenes — and 0 of 20 once the displacement passes 6 cm.
Retraining isn't an acceptable answer in deployment: the base model is a frozen asset, the failure shows up in the customer's scene, and there is no budget for an RL cycle per task. The question we're after is narrower and more useful: can a frozen VLA be recovered using only memory of past episodes, decided at the moment of failure?
The naive answer — retrieve the correction that worked in a similar episode and replay it — fails in an instructive way, and that failure is the spine of this report.
Method
The system does no preventive retrieval. As long as the failure detector hasn't fired, the control path is exactly the base policy's: observation → VLA → action. No memory key is encoded and no bank is queried.
Observation
Cameras + arm proprioception
Base policy
Frozen SmolVLA, 50-step action chunk
Action
7-DoF, OSC control
↓ only if the stall detector fires ↓
Detector
Sustained no-progress, 4 consecutive steps
Retrieval
k=5 from the bank, phase-aligned via embeddings
Retargeting
Waypoints to the live object's frame, XY axes
Closed-loop servo
EEF tracking, gain 30, command capped at 1.0
What object-relative retargeting does
A bank record stores a trajectory that solved a similar scene: its cartesian waypoints, the object's pose at that moment, and the end-effector's pose. The object_relative correction does not replay the stored actions. It takes the approach waypoints, expresses them relative to that episode's object, and re-instantiates them relative to the current scene's object. It then follows them with closed-loop control over the measured end-effector position.
The consequence is geometric: the correction's error doesn't grow with how far the object moved, because the reference moves with the object. That is exactly what a replay-based correction cannot do — reproduce the right action for the wrong scene.
The baseline it's measured against
- fixed_gain — replay of the stored absolute action, blended with the base action by a fixed gain. No training involved, and the stronger of the two replay-style corrections we tested.
- oracle — a privileged OSC controller with ground-truth pose and no policy constraint. It doesn't compete; it sets the ceiling and confirms each test scene is solvable.
On object pose
In simulation, object pose is privileged. To avoid reporting a system that depends on ground truth, every object_relative run injects Gaussian pose noise, σ = 2 cm, before handing the pose to the retargeting step. The numbers below are with that noise applied. It's still an upper bound on what a real perception estimator would give — the main limitation of this report, not a footnote.
Protocol
A split sealed before running, with a hashed manifest, and a paired protocol: every recovery episode is compared against the base episode from the same process, same seed stream. This matters — the base policy is stochastic, and comparing against another run's base produces numbers that don't hold up (§5).
| Benchmark | LIBERO-Object |
| Task | pick_up_the_alphabet_soup_and_place_it_in_the_basket |
| Base policy | HuggingFaceVLA/smolvla_libero (frozen) |
| Perturbation | object pose: dx, dy, dyaw |
| Magnitudes | 1, 2, 3, 4, 5, 6, 8, 10, 12 cm |
| Split | 90 train (bank) / 45 test — 5 per magnitude |
| Split seed | 271 |
| Evaluation seed | 20260727 |
| Manifest SHA-256 | a9987e0a…12cbb |
| Per-episode budget | 400 steps |
| Retrieval | k = 5, embedding phase alignment |
| Pose noise | σ = 2 cm, no bias, no dropout |
| Compute | 1× NVIDIA L4 · 16.3 h for the full batch |
The oracle solved 45 of 45 test scenes, across all nine magnitudes, in a median of 113 steps. The ceiling holds: no test scene is geometrically impossible, so a recovery failure is a failure of the mechanism, not of the test bank.
Results
Across the 45 sealed scenes, object_relative takes success from 4.4% to 46.7%: it rescues 21 episodes the base policy lost and breaks 2 it had won (exact McNemar, p = 0.0001).
Recovery against the paired base, by perturbation magnitude
Fig. 1
Success rate over 5 episodes per magnitude. The base run is the same process and the same seed stream as recovery.
The base policy only wins isolated episodes below 5 cm and collapses to zero from 6 cm on. Recovery stays between 20% and 60% across the whole range, with no downward trend.
View data table
| Magnitude | n | Base | object_relative | Oracle |
|---|---|---|---|---|
| 1 cm | 5 | 0 · 0% | 3 · 60% | 5 · 100% |
| 2 cm | 5 | 0 · 0% | 2 · 40% | 5 · 100% |
| 3 cm | 5 | 1 · 20% | 2 · 40% | 5 · 100% |
| 4 cm | 5 | 0 · 0% | 2 · 40% | 5 · 100% |
| 5 cm | 5 | 1 · 20% | 1 · 20% | 5 · 100% |
| 6 cm | 5 | 0 · 0% | 3 · 60% | 5 · 100% |
| 8 cm | 5 | 0 · 0% | 3 · 60% | 5 · 100% |
| 10 cm | 5 | 0 · 0% | 3 · 60% | 5 · 100% |
| 12 cm | 5 | 0 · 0% | 2 · 40% | 5 · 100% |
| Total | 45 | 2 · 4.4% | 21 · 46.7% | 45 · 100% |
The magnitude cliff, and why it's a replay artifact
The comparison against the replay-based correction is where the result gets interesting. fixed_gain is excellent near the origin — 100% at 1 and 2 cm, better than object_relative — and then falls off a cliff down to 0% at 10 cm. object_relative is the only arm whose curve has no slope.
Two corrections across the perturbation range
Fig. 2
Same 45 scenes, same training bank, same step budget. The x-axis isn't linear — it's the nine magnitudes tested.
The crossover sits between 5 and 6 cm. Past 6 cm, fixed_gain is indistinguishable from doing nothing (3/20 against a 2/20 base, p = 1.0) while object_relative rescues 11 of 20 without breaking a single one.
View data table
| Magnitude | n | object_relative | fixed_gain |
|---|---|---|---|
| 1 cm | 5 | 3 · 60% | 5 · 100% |
| 2 cm | 5 | 2 · 40% | 5 · 100% |
| 3 cm | 5 | 2 · 40% | 3 · 60% |
| 4 cm | 5 | 2 · 40% | 2 · 40% |
| 5 cm | 5 | 1 · 20% | 3 · 60% |
| 6 cm | 5 | 3 · 60% | 1 · 20% |
| 8 cm | 5 | 3 · 60% | 1 · 20% |
| 10 cm | 5 | 3 · 60% | 0 · 0% |
| 12 cm | 5 | 2 · 40% | 1 · 20% |
| Total | 45 | 21 · 46.7% | 21 · 46.7% |
A methodological asymmetry worth stating
The two arms tie in aggregate (21/45 each) but did not run under equivalent conditions. fixed_gain was evaluated with the detector forced always-on, so it took control from step 0 and never had to recover from anything: it's assisted execution from the start. object_relative ran with the real detector, which fired at a median of step 181 — it recovered from a state already degraded by 181 steps of failure. The aggregate tie favors object_relative under a strictly harder condition.
The two regimes, with confidence intervals
Grouping into near field (1–5 cm, n=25) and far field (6–12 cm, n=20), the separation is statistically real in both directions: paired per episode, object_relative loses the near field against fixed_gain (1 vs. 9 discordant episodes) and wins the far field (9 vs. 1), exact McNemar p = 0.0215 in each case.
Success rate by regime, with 95% Wilson intervals
Fig. 3
Near field 1–5 cm (n=25) and far field 6–12 cm (n=20). The intervals are wide: this is one task, one seed.
In the far field, object_relative's interval (34–74%) touches neither the base's (0–16%) nor fixed_gain's (5–36%).
| Correction | Base | Recovery | 95% CI | Rescued | Broke | p |
|---|---|---|---|---|---|---|
| object_relative | 2/45 · 4.4% | 21/45 · 46.7% | 32.9 – 60.9 | 21 | 2 | 0.0001 |
| fixed_gain | 7/45 · 15.6% | 21/45 · 46.7% | 32.9 – 60.9 | 18 | 4 | 0.0043 |
| oracle (ceiling) | — | 45/45 · 100% | 92.1 – 100 | — | — | — |
When recovery happens, and how long it takes
Deployment isn't just about recovering — it has to happen inside the task budget. The detector fires at a median of step 181 out of 400, and when recovery succeeds, it completes in 110 steps, ranging from 92 to 126. That narrow spread is the most useful number for an integrator: the mechanism has no long tail of slow recoveries — it either converges in roughly 110 steps or it doesn't converge at all.
Anatomy of a recovered episode
Fig. 4
Medians over the 21 episodes object_relative recovered, against the 400-step budget.
The detector fires anywhere from step 9 to step 269 depending on the episode. Recovery ends at a median of step 291, leaving 109 steps of margin against the budget.
What this shows, and what it doesn't
The result is a genuine signal from a disciplined protocol, on one task. It is not a validated method. The limitations below aren't formalities — they're the list of reasons we don't yet call this a publishable result at a robotics venue.
What holds
- Recovery > base, sealed, paired, and significant with no training at all (p = 0.0001), against a real VLA inside the control loop.
- Degradation with perturbation magnitude is an artifact of replay-based correction, not a limit of the memory-based approach: object_relative's curve is flat exactly where the replay curve falls.
- The result holds with σ = 2 cm of injected pose noise.
What doesn't hold yet
- One task, one seed, 5 episodes per cell. The per-regime confidence intervals are 20–40 points wide. This is a case study, not a measure of generality.
- Object pose is still privileged. The 2 cm of noise simulates an estimator; it isn't one. The strong version of this claim needs pose estimation from camera images.
- Detector selectivity is unmeasured. It fired in 100% of episodes across both arms, because all 45 test scenes are perturbed. Without nominal episodes in the split there is no false-positive rate — and when it fired on an episode the base was going to win, the correction destroyed it 2 out of 2 times.
- The base varies across runs when the detector configuration changes. Two runs measured a base of 2/45; a third measured 7/45, with 7 discordant episodes. Runs with the same detector are identical to each other, so the suspect is a shift in SmolVLA's stochastic sampling stream. This doesn't affect the paired results inside a single run — which is where every number in this report comes from — but it's an open reproducibility question.
- No comparison against a retrained policy in this sealed split. The RL-trained residual reference arm did not run in this batch.
On novelty
Executing a demonstration relative to the object's frame sits close to one-shot imitation and classic pose-based manipulation. We don't claim “servo to the object” as novel. What this work contributes is empirical: that the replay-based magnitude cliff is removable without any training, and that a detector → retrieval → retargeting layer over a frozen, real VLA recovers episodes the policy loses.
Next steps
Ordered by leverage: first, what could invalidate the story; after that, what extends it.
- Diagnose the cross-run base variance. Without a stable base, the paired protocol isn't reportable beyond a single run.
- Generality before anything else: 3 to 5 tasks across at least two LIBERO suites. It's the first thing a reviewer will ask for, and it's what turns a case study into a method. The per-task geometry tooling already exists.
- Multi-seed evaluation — at least 3 evaluation seeds, with means, confidence intervals, and a paired test per claim.
- Measure detector selectivity by adding nominal, unperturbed episodes to the split, to quantify false positives and the damage to episodes the base already solved.
- Real pose estimation from camera images, to close the gap between the upper bound reported here and a deployable system.
- Sweep pose noise, bias, and dropout beyond the single σ = 2 cm evaluated here.
If you want to receive an update when the open source library is published, sign up for the waitlist.