Orchestrations v1: a public architecture for observable model-tool runs

Research summary

We designed /v1/orchestrations as a public protocol for agent runtimes. A model request, tool discovery, tool calls, tool results, reasoning increments, and final output are represented as a stream-consumable orchestration object. This note records the public contract, the event projection, and the limited performance evidence available at publication.

The protocol is not a claim that a wrapper makes a model intrinsically faster. Its job is to consolidate authentication, policy, routing, metering, and logging boundaries across model providers. Provider queueing, prefill, and first-token decoding remain governed by the model service, request shape, cache state, and network path.

1. Public protocol boundary

The public surface consists of POST /v1/orchestrations, POST /v1/orchestrations/compact, and a WebSocket session on the same path. The primary object uses the orch_ prefix and the orchestration type, with the states queued, in_progress, completed, failed, and incomplete. These identifiers are a client contract; they must not expose provider, region, account, or internal runtime topology.

Inputs cover the model, instructions, input, tools, tool choice, parallel tool calls, reasoning, storage, streaming, inclusion options, service tier, prompt-cache hint, text options, and client metadata. Public metadata is normalized before it reaches the contract. Authorization context, proxy requests, sensitive fields, and runtime log objects are not model input and are not returned in the public response.

2. Streaming events and tools

Both SSE event: values and JSON type values use the orchestration.* namespace. Before final text, a client can observe creation, output-item creation and completion, text deltas, tool-argument deltas and completion, reasoning-summary deltas, metadata, and terminal events. Providers need not offer every event class; the stable promise is the event meaning and recovery behavior, not artificial equivalence between providers.

The tool surface supports functions, namespaces, tool_search, custom tools, and custom call/result items. Namespaces group capabilities by domain. tool_search permits discovery before delayed injection, reducing initial schema pressure. MCP may supply a catalogue or execution backend, while Orchestrations owns the state of one model-and-tool turn and its user-facing event projection.

3. Latency model and observations

We separate time to first visible text into ingress, authentication and policy, normalization, routing, provider queueing, model prefill, first decode, stream transformation, and network delivery. The protocol’s direct additions are normalization and event projection. They are expected to scale with request and event size, but they remain measurements to make—not a basis for an unqualified latency claim.

SSE enables rendering before a response is complete; it does not promise lower total completion time. tool_search trades a discovery turn for a smaller initial declaration. A prompt-cache key is a routing hint rather than a guaranteed hit. Compaction can reduce later prefill work while consuming a service or model budget itself. Comparisons therefore need the same model, region, concurrency, token budget, and cache conditions.

4. Local data-plane measurement

On 14 July 2026, we issued six serial, short, tool-free requests to vauix/deepseek-v4-flash: three JSON and three SSE. The path was the primary node’s local LiteLLM data plane, bypassing public domains, edge gateways, and cross-host networking. The observation establishes that the local protocol path completed an event loop; it is not a production SLA or a percentile study.

All JSON requests succeeded. All SSE requests completed the common event sequence, including creation, progress, output/content-item updates, text deltas, corresponding done events, and completion. The small sample showed earlier observable SSE increments but also meaningful run-to-run variation. We report it as connectivity and instrumentation evidence only.

5. Relationship to other interfaces

Object lifecycles, typed SSE events, function and custom tools, parallel calls, cache hints, service tiers, and storage controls have useful precedents in established model APIs. We treat them as compatibility lessons, not as a hidden dependency: the public object, event namespace, and release cadence remain Orchestrations’ own. Message-oriented and function-calling APIs can be routed behind the same control plane; MCP is complementary rather than competitive.

6. Reliability and security boundary

Clients must handle failed and incomplete; absence of an error event is not success. Whether a tool call can be retried depends on call identity, idempotency, and host policy. Namespaces and delayed discovery reduce exposure, but do not replace identity checks, authorization, parameter validation, audit, or least privilege. Additive event evolution and ignoring unknown events preserve client resilience; removals and semantic changes require versioning or a deprecation period.

7. Verification status

Implementation records include regression assertions for object and event mapping, reasoning-summary indexing, route classification, internal-metadata removal, tool-message ordering, and tool-search materialization. We did not count the local Python test suite as newly rerun: the local virtual environment failed before collection because the standard-library encodings module was unavailable. Publication therefore distinguishes recorded assertions from a clean re-verification.

Next, we will run conformance fixtures for JSON, SSE, and WebSocket in a locked environment; test cancellation and broken streams; test tool idempotency and permission denials; and publish aggregated performance data under fixed conditions.

8. Conclusion

Orchestrations v1 is an observable protocol for a model-and-tool turn. The gateway absorbs provider variation, the runtime normalizes inputs and tool flow, and clients consume stable objects and events. Local observations show that both JSON and SSE paths complete and that SSE can expose text earlier. They do not establish global throughput, latency, or cost superiority.