# API
URL: /docs/api

Reference for New Theory's Inference API



The New Theory API is how your code calls New Theory's hosted robotics models. You send the robot's current state and camera images; the API returns a chunk of target poses. Tasks and observations stream over WebSocket. When the task ends, the connection closes.

The wire schema below is illustrated with a single-arm joint-space contract as a worked example. Shapes and camera keys vary by model — always resolve the contract from the registry (see [Model discovery](#model-discovery)) rather than hardcoding them.

## How the API Works [#how-the-api-works]

<Callout type="info" title="Use the SDK for Python">
  Most developers don't write this protocol by hand. The New Theory SDK wraps the WebSocket, framing, and loop. See the [New Theory SDK](/docs/newt-sdk). Use this page if you're writing a custom client, hitting the endpoint from a non-Python language, or debugging the SDK.
</Callout>

A persistent network connection (WebSocket) carries binary messages encoded as msgpack-numpy — a compact binary format for NumPy arrays.

```python
import asyncio
import os

import msgpack
import msgpack_numpy
import numpy as np
import websockets

msgpack_numpy.patch()

# The inference endpoint for this model, resolved from the registry —
# see Model discovery below.
URL = "wss://<endpoint-from-registry>/stream"


async def main():
    # Authenticate with your API key (loaded from a shell env var).
    headers = {"Authorization": f"Bearer {os.environ['NT_API_KEY']}"}

    # Open one WebSocket per task. The handshake auth happens here.
    async with websockets.connect(URL, extra_headers=headers) as ws:
        # An observation describes the robot's current state for the model.
        obs = {
            "type": "obs",
            # 6 joint values: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper.
            "state": np.zeros(6, dtype=np.float32),
            # Both cameras are expected: two RGB views of the workspace.
            "images": {
                "top":  np.zeros((3, 224, 224), dtype=np.uint8),
                "side": np.zeros((3, 224, 224), dtype=np.uint8),
            },
            "prompt": "pick up the cup",   # plain-English task description
            "max_duration": 5.0,           # wall-clock cap in seconds; first frame only
        }
        # Encode the dict as a binary msgpack message and send it.
        await ws.send(msgpack.packb(obs, use_bin_type=True))

        # Receive one binary message back, decode it, pull the action chunk.
        frame = msgpack.unpackb(await ws.recv(), raw=False)
        chunk = frame["chunk"]  # (30, 6) float32 — 30 target joint positions


asyncio.run(main())
```

The full control loop reads fresh state and sends the next `obs` after executing some prefix of the chunk. See [What you get back](#what-you-get-back).

## Model discovery [#model-discovery]

Inference endpoints are resolved from the model registry, not hardcoded. The first call a custom client makes is `GET /v1/models` against the always-on registry — the same call the SDK makes at `Robot()` construction:

```bash
curl -H "Authorization: Bearer $NT_API_KEY" \
  https://nt-registry-production.up.railway.app/v1/models
```

The response is a JSON list with one object per model: its immutable `uid`, the `tags` you pass to `Robot(model=...)`, the `endpoint` to open the WebSocket against, and the `contract` the server enforces on your frames. Trimmed to one model:

```json
[
  {
    "uid": "ft_base_example",
    "tags": ["<model-tag>"],
    "type": "base",
    "base": null,
    "endpoint": "wss://<endpoint-from-registry>/stream",
    "contract": {
      "state_shape": [6],
      "state_dtype": "float32",
      "cameras": {
        "required": [],
        "expected": ["top", "side"]
      },
      "image_shape": [3, 224, 224],
      "action_shape": [30, 6],
      "action_axes": ["shoulder_pan", "shoulder_lift", "elbow_flex", "wrist_flex", "wrist_roll", "gripper"]
    }
  }
]
```

`GET /v1/models/{uid_or_tag}` returns a single entry. Fine-tune entries inherit `endpoint` and `contract` from their base; the registry returns them resolved. The registry runs on an always-on service, so discovery answers in under a second even when the GPU container serving inference is cold.

## Endpoint and auth [#endpoint-and-auth]

```
wss://<endpoint-from-registry>/stream
```

This is the `endpoint` value the registry returns for the model you resolve. Resolve it from the registry rather than hardcoding it — the value changes as serving infrastructure moves.

Authenticate once, at the handshake:

```
Authorization: Bearer nt_a1b2c3d4...
```

The server verifies against the [console](/docs/authentication). On success, the key is trusted for the connection lifetime — no mid-stream re-auth. On failure, the server closes with code `4001`.

## What you send [#what-you-send]

The client drives inference by sending `obs` frames. Each one carries the robot's current state, camera images, and the task prompt.

```python
{
    "type": "obs",
    "state":  np.ndarray,          # float32 (6,) — joint positions, radians
    "images": {
        "top":  np.ndarray,        # uint8 (3, 224, 224) — CHW, RGB
        "side": np.ndarray,        # uint8 (3, 224, 224)
    },
    "prompt": "pick up the cup",
    "max_duration": 30.0,          # seconds; first frame only
}
```

`state` is the robot's joint positions, in the order the model's `action_axes` declares — for this worked example, shoulder\_pan, shoulder\_lift, elbow\_flex, wrist\_flex, wrist\_roll, then a normalised gripper value.

`images` are CHW uint8 at 224×224. Resize on the client to match — the server's transform pipeline expects this size, and full-resolution camera payloads exceed the connection's message size cap.

Send only the cameras the model expects. An entry for a camera the model doesn't expect — for example, a second arm's wrist camera left in your config file — fails inference server-side. Filter your config down to the model's camera keys before sending.

`max_duration` is read from the first `obs` only; ignored on later frames.

To stop a run early, send:

```python
{"type": "stop"}
```

## What you get back [#what-you-get-back]

The server emits one `action` frame per inference cycle:

```python
{
    "type": "action",
    "chunk": np.ndarray,   # float32 (30, 6)
}
```

Each row of the chunk is a target joint position in the same format as `state`: `[shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper]`. The action horizon is 30 steps.

Clients typically don't execute all 30 steps before requerying. The common pattern is receding-horizon: take a prefix — `max_actions_per_chunk=15` is typical — at a steady cadence, often 15 Hz with `goal_time_s` smoothing between consecutive targets, then read fresh state and send the next `obs`. The model is trained for chunk overlap; tune the cut-off to your control loop.

## How runs end [#how-runs-end]

When the run ends, the server emits one `terminal` frame and then closes the connection:

```python
{
    "type": "terminal",
    "stop_reason": "max_duration",
}
```

| Value          | Meaning                                                                                                |
| -------------- | ------------------------------------------------------------------------------------------------------ |
| `max_duration` | Wall-clock time since the first `obs` exceeded `max_duration`. The expected outcome on current models. |
| `interrupted`  | Client sent a `stop` frame.                                                                            |

Missing or wrong-shape fields produce degraded actions, not a dropped connection. The only hard reject at the frame level is an undecodable msgpack payload.

## Debugging connection issues [#debugging-connection-issues]

When something is wrong at the transport layer — auth, malformed frames, server errors — the WebSocket closes with one of these codes:

| Code   | Meaning                                                     | SDK exception           |
| ------ | ----------------------------------------------------------- | ----------------------- |
| `1000` | Normal closure after a `terminal` frame.                    | —                       |
| `4001` | Auth failed — missing, malformed, or revoked key.           | `AuthError`             |
| `4400` | Protocol error — undecodable frame or missing `type`.       | `ProtocolError`         |
| `4403` | Key is valid but doesn't own the requested model.           | `ForbiddenError`        |
| `4404` | Model identifier not found in the registry.                 | `ModelNotFoundError`    |
| `4422` | An `obs` frame doesn't match the model's declared contract. | `ContractMismatchError` |
| `4500` | Server error outside the model call.                        | `ServerError`           |
| `4503` | Verifier unavailable at handshake.                          | `VerifierError`         |

Errors inside the model close abnormally too: the server sends a `server.inference_error` envelope and closes `4500`. See [Errors](/docs/api/errors) for the envelope shape.

To confirm the URL is reachable and your key works without writing client code, hit the handshake with `websocat`:

```bash
websocat \
  -H "Authorization: Bearer $NT_API_KEY" \
  --binary \
  wss://<endpoint-from-registry>/stream
```

A valid key holds the connection open; a bad one closes `4001`. `websocat` doesn't encode msgpack, so this verifies the handshake but won't drive inference.

## What might change [#what-might-change]

* **Future single-arm joint-space models** will share this wire shape — 6D joint state, two named cameras, `(30, 6)` action chunks. A client written against this page should keep working as the New Theory model family advances.
* **Other models during early development.** We serve MolmoAct2 and potentially other models to prototype the API with real developers; some may use different wire shapes. The long-term direction is one consistent wire across models; in the short term, expect shifts.