API

The New Theory API is how your code calls New Theory's hosted robotics models. You send the robot's current state and camera images; the API returns a chunk of target poses. Tasks and observations stream over WebSocket. When the task ends, the connection closes.

The wire schema below is illustrated with a single-arm joint-space contract as a worked example. Shapes and camera keys vary by model — always resolve the contract from the registry (see Model discovery) rather than hardcoding them.

How the API Works

A persistent network connection (WebSocket) carries binary messages encoded as msgpack-numpy — a compact binary format for NumPy arrays.

import asyncio
import os

import msgpack
import msgpack_numpy
import numpy as np
import websockets

msgpack_numpy.patch()

# The inference endpoint for this model, resolved from the registry —
# see Model discovery below.
URL = "wss://<endpoint-from-registry>/stream"


async def main():
    # Authenticate with your API key (loaded from a shell env var).
    headers = {"Authorization": f"Bearer {os.environ['NT_API_KEY']}"}

    # Open one WebSocket per task. The handshake auth happens here.
    async with websockets.connect(URL, extra_headers=headers) as ws:
        # An observation describes the robot's current state for the model.
        obs = {
            "type": "obs",
            # 6 joint values: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper.
            "state": np.zeros(6, dtype=np.float32),
            # Both cameras are expected: two RGB views of the workspace.
            "images": {
                "top":  np.zeros((3, 224, 224), dtype=np.uint8),
                "side": np.zeros((3, 224, 224), dtype=np.uint8),
            },
            "prompt": "pick up the cup",   # plain-English task description
            "max_duration": 5.0,           # wall-clock cap in seconds; first frame only
        }
        # Encode the dict as a binary msgpack message and send it.
        await ws.send(msgpack.packb(obs, use_bin_type=True))

        # Receive one binary message back, decode it, pull the action chunk.
        frame = msgpack.unpackb(await ws.recv(), raw=False)
        chunk = frame["chunk"]  # (30, 6) float32 — 30 target joint positions


asyncio.run(main())

The full control loop reads fresh state and sends the next obs after executing some prefix of the chunk. See What you get back.

Model discovery

Inference endpoints are resolved from the model registry, not hardcoded. The first call a custom client makes is GET /v1/models against the always-on registry — the same call the SDK makes at Robot() construction:

curl -H "Authorization: Bearer $NT_API_KEY" \
  https://nt-registry-production.up.railway.app/v1/models

The response is a JSON list with one object per model: its immutable uid, the tags you pass to Robot(model=...), the endpoint to open the WebSocket against, and the contract the server enforces on your frames. Trimmed to one model:

[
  {
    "uid": "ft_base_example",
    "tags": ["<model-tag>"],
    "type": "base",
    "base": null,
    "endpoint": "wss://<endpoint-from-registry>/stream",
    "contract": {
      "state_shape": [6],
      "state_dtype": "float32",
      "cameras": {
        "required": [],
        "expected": ["top", "side"]
      },
      "image_shape": [3, 224, 224],
      "action_shape": [30, 6],
      "action_axes": ["shoulder_pan", "shoulder_lift", "elbow_flex", "wrist_flex", "wrist_roll", "gripper"]
    }
  }
]

GET /v1/models/{uid_or_tag} returns a single entry. Fine-tune entries inherit endpoint and contract from their base; the registry returns them resolved. The registry runs on an always-on service, so discovery answers in under a second even when the GPU container serving inference is cold.

Endpoint and auth

wss://<endpoint-from-registry>/stream

This is the endpoint value the registry returns for the model you resolve. Resolve it from the registry rather than hardcoding it — the value changes as serving infrastructure moves.

Authenticate once, at the handshake:

Authorization: Bearer nt_a1b2c3d4...

The server verifies against the console. On success, the key is trusted for the connection lifetime — no mid-stream re-auth. On failure, the server closes with code 4001.

What you send

The client drives inference by sending obs frames. Each one carries the robot's current state, camera images, and the task prompt.

{
    "type": "obs",
    "state":  np.ndarray,          # float32 (6,) — joint positions, radians
    "images": {
        "top":  np.ndarray,        # uint8 (3, 224, 224) — CHW, RGB
        "side": np.ndarray,        # uint8 (3, 224, 224)
    },
    "prompt": "pick up the cup",
    "max_duration": 30.0,          # seconds; first frame only
}

state is the robot's joint positions, in the order the model's action_axes declares — for this worked example, shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, then a normalised gripper value.

images are CHW uint8 at 224×224. Resize on the client to match — the server's transform pipeline expects this size, and full-resolution camera payloads exceed the connection's message size cap.

Send only the cameras the model expects. An entry for a camera the model doesn't expect — for example, a second arm's wrist camera left in your config file — fails inference server-side. Filter your config down to the model's camera keys before sending.

max_duration is read from the first obs only; ignored on later frames.

To stop a run early, send:

{"type": "stop"}

What you get back

The server emits one action frame per inference cycle:

{
    "type": "action",
    "chunk": np.ndarray,   # float32 (30, 6)
}

Each row of the chunk is a target joint position in the same format as state: [shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper]. The action horizon is 30 steps.

Clients typically don't execute all 30 steps before requerying. The common pattern is receding-horizon: take a prefix — max_actions_per_chunk=15 is typical — at a steady cadence, often 15 Hz with goal_time_s smoothing between consecutive targets, then read fresh state and send the next obs. The model is trained for chunk overlap; tune the cut-off to your control loop.

How runs end

When the run ends, the server emits one terminal frame and then closes the connection:

{
    "type": "terminal",
    "stop_reason": "max_duration",
}

Missing or wrong-shape fields produce degraded actions, not a dropped connection. The only hard reject at the frame level is an undecodable msgpack payload.

Debugging connection issues

When something is wrong at the transport layer — auth, malformed frames, server errors — the WebSocket closes with one of these codes:

Errors inside the model close abnormally too: the server sends a server.inference_error envelope and closes 4500. See Errors for the envelope shape.

To confirm the URL is reachable and your key works without writing client code, hit the handshake with websocat:

websocat \
  -H "Authorization: Bearer $NT_API_KEY" \
  --binary \
  wss://<endpoint-from-registry>/stream

A valid key holds the connection open; a bad one closes 4001. websocat doesn't encode msgpack, so this verifies the handshake but won't drive inference.

What might change

• Future single-arm joint-space models will share this wire shape — 6D joint state, two named cameras, (30, 6) action chunks. A client written against this page should keep working as the New Theory model family advances.
• Other models during early development. We serve MolmoAct2 and potentially other models to prototype the API with real developers; some may use different wire shapes. The long-term direction is one consistent wire across models; in the short term, expect shifts.