Two clocks — action rate vs inference rate

A common confusion: the API is on a streaming connection, so it sounds like we're streaming inference at the arm's control rate. We aren't. The arm and the model run on two different clocks, decoupled by the model's prediction horizon.

The two clocks

• Arm clock: ~15 Hz — every 67 ms, the arm gets a new target pose and moves toward it.
• Inference clock: ~0.4 Hz — every ~2.5 s, the SDK sends an observation and the cloud returns a plan.

Inference is roughly 40× slower than the arm's natural cadence. The two have to be reconciled somehow.

How they're reconciled — the model emits plans, not actions

The model doesn't return one action. It returns a plan: 30 future poses, covering roughly 2 s of intended motion (30 actions at the arm's ~67 ms step). The arm plays that plan back at 15 Hz over the same ~2 s of wall time.

While the arm is mid-plan, the SDK kicks off the next inference call. By the time the arm has played the first ~20 actions of the current plan (~1.3 s of motion), the next plan is arriving from the cloud. The SDK swaps to it; the arm starts playing actions from the new plan.

T=0.00 s   send obs A  ────────┐
T=0.00 s   start playing plan_prev[0..19]   (~1.3 s of motion)
T=1.33 s   discard plan_prev[20..30]; we want fresh
T=2.50 s   plan A arrives ─────┘
T=2.50 s   start playing plan A[0..19]
T=3.83 s   discard A[20..30]; send obs B
...

The arm moves smoothly the whole time. The model only "sees" the robot every ~2.5 s — not every 67 ms.

What this means practically

The model has to plan, not react. A single-action API would be useless here: the action returned would be 2.5 s stale by the time it executed. So the model emits forward-looking plans — predicting what to do over the next 1–2 seconds based on the latest observation it has.

Reactive correction has a floor of ~1 second. Between observation round-trips, the arm is executing the model's plan blindly. If something unexpected happens (the object slips, you bump the arm), the model can't notice until the next observation arrives. Hardware-level reflexes (collision detection, joint velocity limits) catch the dangerous cases; the model isn't fast enough.

Plans need to be good. Since the arm is flying blind for ~1 second between observations, the model can't emit drunk-walk trajectories that need constant correction. It has to predict competently far enough ahead that the arm tracks something useful even with stale conditioning.

Receding horizon discards work to stay fresh. The model gives the SDK 30 actions; the demo applies ~20 and throws away the rest. Why discard? Because by action 20, the observation that produced the plan is already 1.3 s old. Asking "what should I do now given where I am now" with a fresh observation beats trusting the older plan's tail.

What "streaming" actually buys us here

It's not that we're streaming frames at the arm's rate. What we get from the streaming connection:

1. One handshake per session, not per inference call. Even at 0.4 Hz, handshake-per-request would add ~150 ms of TCP + TLS overhead to every cycle. The persistent connection amortizes that to one cost per run().
2. A bidirectional channel for out-of-band signals. The model can push a terminal frame when the task is complete; the SDK can push a stop frame when the operator aborts. No polling, no separate endpoints.
3. A session shape that matches the work. One robot trial = one WebSocket = one run() call. The conversation has a defined start and end built into the protocol.

The fundamental trade

We get the responsiveness of high-frequency arm control (15 Hz, smooth motion, tight joint velocity limits) and the smartness of slow cloud inference (a big vision-language-action model, language conditioning, generalization across tasks). The cost is the ~1 second of open-loop drift between observations.

Streaming doesn't eliminate that — nothing eliminates that until inference gets much faster. What streaming does is minimize the protocol overhead so the inference clock can run as fast as the model itself allows.

So the precise mental model: the SDK plan-streams, not inference-streams. The model emits a plan; the arm consumes that plan at its own rhythm; the SDK re-plans periodically using a fresh observation. The streaming connection is the conduit that makes that re-plan cycle as cheap as possible.

Related

• WebSocket vs HTTP — why we picked a streaming connection at all