Vtslx AO cache architecture and semantics

Research summary

This note documents the public semantics of caching in Vtslx AO. We treat caching as an optimization layer, never as a correctness dependency: a miss, read failure, eligibility rejection, or semantic invalidation returns to the same authorized execution path. The research focus is therefore not a hit-rate headline, but the conditions under which reuse is safe, observable, and reversible.

1. Goals and principles

AI gateway requests contain mutable authorization context, tools with side effects, streaming state, and sensitive content. Reusing an answer across any of those boundaries can be a data or correctness failure. We began from four principles: bind reuse to tenant and policy scope; separate gateway coordination from model-provider reuse; never replay side effects merely because a key matches; and expose fallback and eligibility outcomes as operational signals.

2. Public reference architecture

The reference path first normalizes the request, evaluates policy and cache eligibility, then enters the orchestration runtime. Gateway state may coordinate short-lived health and quota observations. A model route may receive an eligible context-prefix hint. Neither layer turns a cache into an alternative authorization path.

The design deliberately does not disclose internal cache-key material, TTLs, capacity settings, or infrastructure topology. Those values are operational controls, not a public interoperability contract.

3. Cached objects and eligibility

Eligibility is a vector rather than a boolean: tenant, authorization, normalized input, model route, policy revision, data version, tool behavior, output form, and requested freshness can all change its result. Deterministic, short requests may qualify for an explicitly approved final-result reuse policy. A partial stream cannot. A tool call with side effects cannot. Health observations may be coordinated briefly, but are never a permanent availability promise.

This separation protects a useful distinction: gateway state helps the system avoid a thundering herd, while a provider’s prefix cache may lower repeated context processing. Neither is evidence that an entire user-visible answer can safely cross a session boundary.

4. Layer semantics and fallback

The lifecycle starts with semantic normalization, not a raw request hash. The eligibility gate then selects an allowed read or coordination path. Only a complete, authorized terminal result can be written after integrity checks. The same policy and routing chain handles every miss and invalidation.

This ensures that cache unavailability degrades performance rather than behavior. It also keeps the analysis of a fast response honest: lower latency can originate from a prefix hit, a warm connection, a different provider queue, or a smaller input—not necessarily from a shared final result.

5. Tools, streaming, and consistency

In-progress SSE or WebSocket events remain owned by the initiating turn. They are not inventory to resell to another session. Tool invocations are non-shareable by default because a call can depend on permissions, external state, and side effects. A narrow policy may cache an authorized summary from a demonstrably side-effect-free tool, bound to source scope and data version; it must not make writes, payment, deployment, or permission changes replayable.

Consistency also has to be explicit. Revoking access, changing a policy, updating source data, or changing a model route invalidates a reuse claim even where bytes happen to look identical.

6. Metrics and governance

We measure cached-prefix tokens separately from a full-answer hit, record the rate at which policy correctly rejects unsafe eligibility, and measure fallback success when caches are unavailable. The difference between time-to-first-byte or first-token under controlled conditions can be informative, but it is not portable across models, regions, concurrency, or inputs.

Governance covers scope binding, access-revocation invalidation, minimized values, version namespaces, audit, rate controls, and safe fallback. A higher rejection rate can be evidence of a working safety boundary rather than a regression.

7. Conclusion and roadmap

Vtslx AO caches only when an authorized, semantically stable reuse case is demonstrated. It separates gateway coordination, provider prefix hints, client state, and any approved result reuse; it refuses to make streams or side-effecting tools shared artifacts. Future work will add controlled measurements, policy-version visibility, and stronger source-version contracts without turning cache behavior into a correctness prerequisite.