PRISM: Polynomial Representations for Interaction-Structured Motor Control

Motivation

A joint value is only meaningful in context.

joint angle×velocity

distinguishes a static pose from moving momentum

command×realized motion

reveals slip and dynamics mismatch

motion×slowdown

provides a proprioceptive cue for interaction

Method

PRISM starts near-linear, then learns useful quadratic context.

The learned feature-wise scale α is initialized to 0.01. Training can preserve the linear path or amplify the second-order factor where it helps.

Deployable input Proprioceptive history x

jointsvelocitiesactionscommands

Linear factor u = W_lx

Learned context c = 1 + α ⊙ W_rx

⊙

GatedQuad feature z = u ⊙ c linear behavior + learned degree-2 interactions

Policy token φ(x) MLP → RMSNorm → unchanged backbone

Near-linear initialization stabilizes optimization. Factorized products avoid an explicit x ⊗ x expansion. Same interface isolates representation from sensing and control.

Headline results

Interaction structure beats simply making the conditioner larger.

Matched checkpoints, shared observations, and the same downstream policy.

RL locomotion

BFM-Zero

1.224 mean Tracking EMD

PRISM lowers tracking mismatch in all three dynamics conditions.

Method Nominal Low friction Payload

BFM-Zero 32.109M 1.1041.5821.121

              PRISM 32.665M
              1.0501.5481.073
            

Larger 49.260M 1.0901.5891.114

Tracking EMD ↓ at the aligned 9.6M checkpoint.

VLA manipulation

SmolVLA on LIBERO

66.55 average success

PRISM improves the same VLA backbone with fewer parameters than the larger control.

SmolVLA63.50

450.046M parameters

PRISM66.55
451.925M parameters

Larger64.90

456.245M parameters

Official LIBERO multi-task eval50 at 80K.

BFM-Zero / locomotion

PRISM tracks the reference more closely under dynamics shifts.

Select a condition to connect the aggregate metric with its aligned whole-body rollout.

PRISM

Low-friction tracking The hardest dynamics shift in the aligned evaluation.

GT | BFM-Zero | Larger | PRISM

Tracking EMD Low friction

lower is better

-2.1% PRISM vs. BFM-Zero

Reference coverage

One policy follows non-walking whole-body motions.

These clips show motion diversity; aggregate EMD above is the comparison metric.

PRISM

SmolVLA / manipulation

The same interaction model improves a VLA policy.

PRISM changes only proprioceptive conditioning; the frozen SmolVLM and action-expert design remain shared.

LIBERO average success 63.50 → 66.55 +3.05 points over SmolVLA

Spatial70.0

Object57.4

Goal85.4

Long53.4

Same-episode comparisons

PRISM completes episodes where both controls stall.

All three panels use the same task and evaluation episode.

PRISM success

Long-horizon placement PRISM completes the object placement while both controls stall.

SmolVLA | Larger | PRISM

Contact behavior

PRISM keeps moving with lower logged force spikes.

Force is recorded after rollout for analysis and is never provided to the policy.

PRISM success

Force p9516.6PRISM

EEF speed0.34PRISM mean

Policy force inputNoneanalysis only

Synchronized rollout trace Contact force norm

SmolVLA Larger PRISM

Loading trace...

Lower peaks indicate softer contact while the synchronized video shows whether motion continues.

Representation analysis

PRISM makes shifted dynamics easier to separate.

t-SNE of BFM-Zero actor features for the same reference motion. Switch methods to compare how scenario structure changes.

Loading representation...

Takeaway

Expose interactions, keep the robot interface simple.

PRISM adds a compact, learnable degree-2 representation to deployable proprioception. It improves strong RL and VLA backbones without adding sensors, privileged physical parameters, or a separate controller.

Paper PDF arXiv pending Code

Citation

BibTeX

@article{lee2026prism,
  title  = {PRISM: Polynomial Representations for Interaction-Structured Motor Control},
  author = {Lee, Seung Hyun and Yu, Stella X.},
  year   = {2026}
}