Wire Protocol Specification

vgi-rpc is an RPC framework where serialization uses the Apache Arrow IPC Streaming Format.

• All integers are little-endian unless stated otherwise
• All metadata strings are UTF-8 encoded
• Wire protocol version: "1" (ASCII 0x31)
• Metadata keys in the vgi_rpc.* namespace are framework-reserved

Each logical message exchange uses one or more IPC streams written sequentially on the same byte stream (pipe, TCP socket, HTTP body, etc.).

An IPC stream consists of:

1. Schema message — describes the columns and their Arrow types
2. Zero or more RecordBatch messages — each optionally carrying per-batch custom metadata
3. EOS marker — the 8-byte sequence 0xFF 0xFF 0xFF 0xFF 0x00 0x00 0x00 0x00

Multiple IPC streams are written sequentially on the same underlying byte stream. Each reader opens one stream, reads until EOS, and stops. The next reader picks up immediately after.

RPC parameters and return values are serialized as Arrow columns. Cross-language implementations MUST use these Arrow types for interoperability.

A dataclass used directly as an RPC parameter or return value is a binary column containing a complete Arrow IPC stream. Inside that dataclass stream, nested dataclasses are Arrow structs. Container conversion is recursive, including lists, maps, sets, enums, nullable values, and nested dataclasses.

Every RPC request is a single IPC stream containing exactly one batch with one row:

IPC Stream:
  Schema message:
    - One field per method parameter
    - Field types per the type mapping
  RecordBatch message:
    - Exactly 1 row
    - custom_metadata:
        vgi_rpc.method = "<method_name>"       (REQUIRED)
        vgi_rpc.protocol = "<protocol_name>"   (REQUIRED)
        vgi_rpc.request_version = "1"          (REQUIRED)
        vgi_rpc.protocol_version = "1.4.0"     (when declared)
  EOS marker

For methods with no parameters, the schema has zero fields and the batch should represent one row with zero columns. Because Arrow cannot encode a row count for a zero-field batch consistently across runtimes, servers accept any row count for that special case.

A unary response is a single IPC stream on the result schema:

IPC Stream:
  Schema message:
    - Single field named "result" (or empty for void)
  0..N log batches (zero-row, with log metadata)
  1 result or error batch:
    - Result: 1-row batch with return value
    - Void: 0-row batch on empty schema
    - Error: 0-row batch with EXCEPTION metadata
  EOS marker

Log batches MUST appear before the result/error batch.

A log batch is a zero-row batch on the response stream's schema with vgi_rpc.log_level and vgi_rpc.log_message in custom metadata.

When log_level is "EXCEPTION", the client MUST raise/throw an error with:

• error_type: from log_extra.exception_type
• error_message: from vgi_rpc.log_message
• remote_traceback: from log_extra.traceback
• request_id: from vgi_rpc.request_id
• error_kind: stable, open-ended classification — a client MUST treat an unrecognised value as an unclassified error rather than rejecting the batch

On protocol_not_supported a client SHOULD call vgi_rpc.Reflection.v1/list_protocols to learn what the server does host, and MUST surface both what it asked for and what it was told. Section 16 defines five further kinds specific to vgi_rpc.Identity.v1.

Streaming methods use a multi-phase exchange over a bidirectional byte stream.

Phase 1: Request parameters

Identical to a unary request: IPC stream with params schema, 1 request row, EOS.

Phase 1.5: Optional header stream

When the stream method declares a header type, the server sends a header IPC stream (header schema, log batches, 1 header row, EOS) before the main data exchange.

Phase 2: Lockstep data exchange

Producer:                        Exchange:
  Client    →   tick (0-row)       Client    →   input batch
  Server    ←   log* + data        Server    ←   log* + output
  Client    →   tick (0-row)       Client    →   input batch
  Server    ←   log* + data        Server    ←   log* + output
  Client    →   tick               Client    →   [EOS]
  Server    ←   [EOS] (finish)     Server    ←   [EOS]

Lockstep is normative: one client tick or input batch drives exactly one server process() invocation and produces zero or more log batches followed by at most one data batch. A producer that finishes closes its output stream instead of returning a data batch for the final tick.

HTTP follows the same rule. The {prefix}/{protocol}/{method}/init request drives the first producer turn; every continuation request carries one tick and drives exactly one additional turn. The request batch's custom metadata is delivered to that turn after stripping vgi_rpc.stream_state#b64, vgi_rpc.call_state#b64, and vgi_rpc.cancel.

A client may cancel either stream by sending a zero-row input batch carrying vgi_rpc.cancel = "1". Presence of the key is the signal; readers must not depend on its value. Cancellation is best-effort, does not invoke process(), invokes the stream state's cancellation hook, and closes cleanly rather than producing an exception batch.

The HTTP transport maps the pipe-based protocol to stateless HTTP request/response pairs. All requests use Content-Type: application/vnd.apache.arrow.stream.

The first three rows are the RPC surface and are namespaced by protocol; the path segment must agree with vgi_rpc.protocol (Section 3.1). Introspection is not a special path — it is {prefix}/vgi_rpc.Reflection.v1/list_protocols and {prefix}/vgi_rpc.Reflection.v1/describe, reached the same way as anything else, and identity is likewise {prefix}/vgi_rpc.Identity.v1/introspect_token. The reserved framework endpoints are deliberately not namespaced: they belong to the server rather than to any one protocol. A GET to a two-segment path whose first segment cannot be a protocol name is 404, not 405 — without that rule any unrelated two-segment path, a /.well-known/… document among them, matches the RPC route and is answered “method not allowed” by it.

Capability headers are stamped on every response and can be discovered cheaply at HEAD {prefix}/health. Browser clients must use HEAD for discovery so the probe passes the CORS method check. GET and OPTIONS also carry capability headers. There is no __capabilities__ endpoint. An absent capability means unsupported or unconfigured, except that a missing VGI-Supported-Encodings indicates a legacy server for which clients assume zstd; a present but empty value explicitly disables compression.

Clients can send VGI-Accept-Max-Response-Bytes as one ASCII positive decimal integer from 65536 through 2⁵³ − 1. Signs, leading zeroes, comma lists, and duplicate values are invalid and receive HTTP 400 before dispatch. The effective decoded response cap is the minimum of the client limit and the application and hosting caps. A batch still too large after externalization produces a cursor-free ResponseTooLargeError. Each HTTP producer turn invokes the producer once; a response target does not combine multiple turns.

Request and response compression are independent. Requests use Content-Encoding; responses are negotiated with Accept-Encoding or the higher-priority X-VGI-Accept-Encoding. The corresponding response header is Content-Encoding or X-VGI-Content-Encoding. Codec tokens are zstd, gzip, and identity; the portable conformance worker supports zstd and gzip in both directions.

HTTP streams split state into two XChaCha20-Poly1305-sealed tokens: immutable call state minted once by /init, and a per-turn cursor. Clients must echo both on every continuation, exchange, and cancellation request. Servers authenticate the cursor first, use its authenticated call ID for cache lookup, then open the call token on a cache miss and require matching call IDs.

Base64-decoded state token envelope:
  byte 0       independent token version (cursor: 5, call: 1)
  bytes 1..24  random XChaCha20-Poly1305 nonce
  bytes 25..   authenticated ciphertext + 16-byte tag

Sealed payload:
  byte 0       self-describing codec (0x00 raw, 0x01 zstd in reference)
  bytes 1..    framed state, compressed before encryption when smaller

Token AAD binds the caller's domain and principal. Implementations must bound decompression, reject unknown codecs, and return the same HTTP 400 classification for invalid, expired, mismatched, or unresolvable tokens.

Authentication runs before method dispatch. Rejections use HTTP 401 with a machine-readable VGI-Auth-Reason; workers may additionally require a per-request HMAC proxy proof while preserving the caller's end-user credential.

The shared memory side-channel enables zero-copy batch transfer between co-located processes. The pipe carries control messages; large batches are written to shared memory and replaced with pointer batches.

SHM is an optional side channel for pipe, subprocess, and Unix-socket transports. Before advertising a segment or writing a pointer, both peers must negotiate vgi_rpc.transport.shm = "true" through __transport_options__. Missing support or any negotiation error requires an inline fallback.

Segment header (64 KiB):
  Offset  Size    Field
  0       4       magic: "VGIS" (0x56 0x47 0x49 0x53)
  4       4       version: uint32 = 1
  8       8       data_size: uint64
  16      4       num_allocs: uint32
  20      4       padding: uint32 = 0
  24      N*16    allocations: (offset, length) pairs

SHM pointer batches are zero-row batches with vgi_rpc.shm_offset and vgi_rpc.shm_length in custom metadata. The allocator uses first-fit with implicit coalescing. Maximum 4,094 allocations. A resolver validates bounds, opens the IPC stream from the referenced region, retains the mapped owner for the Arrow buffers' lifetime, annotates the resolved batch with vgi_rpc.shm_source, and releases the allocation when it is safe to reuse.

When batches exceed a configurable size threshold, they can be externalized to remote storage (HTTPS, S3, or GCS) and replaced with pointer batches containing a vgi_rpc.location URL.

Writers may attach vgi_rpc.location.sha256; readers that receive it must verify the uncompressed payload. Resolution enforces scheme, host, redirect, encoded-size, decoded-size, and retry limits before opening the fetched IPC stream.

Every request batch MUST carry vgi_rpc.request_version = "1".

Three independent things can be skewed between a client and a worker, and each has its own mechanism. None substitutes for another, and a port that implements two of the three has a silent failure mode.

The third row is the one nothing else covers. Protobuf's field numbers and unknown-field semantics make a renamed or retyped field self-detecting; Arrow has neither. A parameter renamed between releases, or retyped under the same name, produces a request the server will happily coerce through its own declared type and act on. The gate is what turns that into a directional error.

Application protocols may independently declare vgi_rpc.protocol_version as canonical semver — no prereleases, no build metadata. Major and minor must match exactly; patch differences interoperate. A server MUST also reject a request carrying a parameter its protocol does not declare: a strict version gate sitting on a lenient deserializer is two policies in one codepath.

The gate is per binding. A server hosting several protocols has a version per protocol and no single “server version”; gating against the primary rejects correct callers of a secondary and names the wrong protocol when it does, so the error message MUST name the protocol. vgi_rpc.Reflection.v1 is exempt — it is what a version-mismatched client calls to learn what mismatched, and the exemption is a property of the binding, not of a method name.

An application failure never reaches the client as a 5xx: intermediaries and HTTP client libraries routinely discard or replace response bodies on 5xx, and the body is precisely where the typed error lives. Clients MUST therefore treat 200 as “a response arrived”, not “the call succeeded”, and classify by inspecting the body — the batch-classification algorithm in Section 7 already does this, so X-VGI-RPC-Error is a fast path, not a second source of truth.

Introspection is an ordinary co-hosted protocol, not a special method name. That is what lets every port generate it from the same pipeline as any other method rather than hand-maintain a bespoke format — which is how the ports drifted before.

A server that offers introspection hosts vgi_rpc.Reflection.v1 and it appears in that protocol's own output. A server that does not simply does not host it, and a client asking gets the ordinary protocol_not_supported — not a bespoke “introspection is disabled” to special-case.

Methods

list_protocols() -> ProtocolList
describe(protocol: utf8) -> ServiceDescription

list_protocols is the cheap question — what is here, and has it changed — and is the only one a client needs on a warm path, because protocol_hash answers “has it changed” without transferring any schema. describe is the expensive one, asked once. A client that does not already know a protocol name needs both, in that order. Both are ordinary unary methods: they carry vgi_rpc.protocol = "vgi_rpc.Reflection.v1", their replies ride as serialized bytes in a single result column, and a server that externalizes payloads externalizes these too.

Payload

ProtocolList carries server_id, server_version, request_version, and protocols: list<ProtocolSummary> — every hosted protocol, reflection included.

ServiceDescription is those fields plus methods: list<MethodInfo>, sorted by name. It deliberately carries no server identity: two processes serving one protocol must describe it identically, or the description is not a property of the protocol. Server identity lives on ProtocolList, which is a statement about a server.

Schemas travel as serialized Arrow IPC rather than as a structural description: a client's whole purpose in asking is to get a schema it can hand to its own Arrow implementation. An absent schema is empty bytes rather than null, so no port pays a null check on a value it will only ever treat as absent.

Decoding is tolerant — normative

A decoder MUST read fields by name, not by position; ignore columns it does not know; and default columns that are absent and have a default. This is what makes minor skew survivable in both directions, and it is load bearing: a strict decoder fails at exactly the moment a client most needs a good answer. A decoder MUST NOT default a field that has no default — it errors instead. One rule follows, binding every port: a field added in a minor version MUST carry a default, or the addition is a breaking change wearing a minor version number.

protocol_hash — normative

The hash is a fingerprint of a protocol's wire surface, so a client and a worker can say which one they have without transferring the whole description. It is defined over what Arrow decodes to, not what an encoder emits: each language's Arrow implementation may legitimately produce different bytes for the same logical schema.

protocol_hash = lowercase_hex(sha256(
    b"vgi_rpc.protocol_hash.v1|" + canonical_json(description)))

The canonicalisation profile is RFC 8785 (JCS), chosen for its published test vectors. The structure is restricted to objects, arrays, strings, and booleans: every numeric parameter is folded into a type token, so the preimage contains no JSON numbers and JCS's number-canonicalisation rule — its hardest, and the likeliest place for seven ports to diverge — never applies.

{"protocol":"vgi_rpc.Reflection.v1","methods":[
  {"name":"describe","type":"unary","has_return":true,"has_header":false,
   "params":[{"name":"protocol","nullable":false,"type":"utf8"}],
   "result":[{"name":"result","nullable":false,"type":"binary"}]},
  {"name":"list_protocols","type":"unary","has_return":true,"has_header":false,
   "params":[],
   "result":[{"name":"result","nullable":false,"type":"binary"}]}]}

• Methods are sorted by name; a port iterating a hash map must still produce this order.
• Field order within a schema is declaration order and is significant.
• result and header are omitted when the method has none. A method returning nothing is not a method returning an empty struct.
• Server identity, docstrings, parameter defaults, language-specific type names, and request_version are not in the preimage.
• stream_kind is not either, for a different reason: which methods a port can classify depends on how that port's registration works, so two ports can disagree while neither is wrong, and a field one port can state and another cannot is not a contract.
• The v1 in the domain tag is the only version the hash carries, and it moves only when the hash definition moves.

Types appear in the preimage as lowercase ASCII tokens — utf8, decimal128(p,s), timestamp(unit,tz=Z), list<item?:T>, struct<a:T,b?:U> — with parameters in parentheses, children in angle brackets, and a trailing ? on a nullable child. List and map child field names are normalised to item / key / value, because Arrow's own type equality ignores them; everything Arrow does treat as part of the type is kept, child nullability included. A timezone is carried verbatim — UTC and +00:00 are distinct Arrow types. A type with no token MUST raise rather than fall back to the Arrow implementation's own string form. The canonical specification carries the full token table.

protocol_hash is also the access log's registry key for decoding archived records. Each access record carries the owning binding's protocol and protocol_hash — the protocol that owns the dispatched method, not a server-wide default. Framework endpoints owned by no protocol log the server's primary. A mislabelled record is well-formed, passes the schema, and produces a plausible dashboard, so it is the field most worth getting right. Method names may collide across protocols, so anything aggregating on method alone must group by (protocol, method) or it silently merges two protocols' traffic.

Before using the shared-memory side channel over pipe, subprocess, or Unix socket transports, peers negotiate once through the synthetic unary method __transport_options__. It is a framework built-in, resolved before routing and owned by no protocol, so it carries no vgi_rpc.protocol routing key. HTTP deployment capabilities use response headers instead; the discovery probe there is HEAD {prefix}/health.

shm_enabled = client["vgi_rpc.transport.shm"] == "true"
           && server["vgi_rpc.transport.shm"] == "true"

Capabilities use the open-ended vgi_rpc.transport.* namespace. Unknown keys are ignored. If the method is absent, errors, or reports false, the client stays inline.

Two optional methods, hosted as an ordinary co-hosted protocol. This was previously an HTTP-only JSON route, POST {prefix}/__introspect_token__, which meant it existed on one transport only and had to be hand-written in every port.

introspect_token(token: utf8) -> TokenIdentity
issue_grant(purpose: utf8, scopes: list<utf8>, ttl_seconds: int64) -> IssuedGrant

A method whose hook the deployment did not configure is not hosted at all, and the binding's method set — and therefore its protocol_hash — narrows with it. A worker that resolves credentials but does not mint grants hosts introspect_token alone, and a client learns that from reflection rather than by calling and reading an error. Absent beats routed-and-refusing: only a narrowed digest distinguishes a worker that cannot mint from one that will not, and it is what keeps a dependency upgrade from growing a credential-to-identity oracle on every existing worker.

introspect_token — an oracle, and guarded as one

Resolves an opaque bearer credential to the principal it authenticates as, for a reverse proxy that terminates the only public listener and must know the caller's identity before it can authorize anything. The answer is an identity assertion made by the thing being protected, which the asker then acts on using credentials the worker does not hold — so it must be trusted more than the worker, not merely as much. Hence four normative guards:

• An introspector allowlist with no permissive default. A server whose resolver is configured without one MUST refuse to start. “Any authenticated caller” lets any user resolve any other user's credential to its owner.
• Authorization is checked before the credential is touched, so an unauthorized caller learns nothing about it — including how long looking at it took.
• Rejections are uniform. Unknown, expired, malformed, and over-long are one answer; distinguishing them confirms that a guessed credential exists.
• A JWS-shaped subject is refused before the resolver runs. Three dot-separated base64url segments are validated locally against a key set.

Rate limiting (20 per caller per second by default) bounds, rather than closes, the oracle an allowlisted-but-compromised caller still has. TokenIdentity carries principal, token_name, and ttl_seconds, and never claims — a pass-through claims field would let a worker choose its caller's tenant routing, row scope, and policy branch. ttl_seconds is how long the answer may be cached: an authorization window, and therefore the revocation lag.

issue_grant — not an oracle, so not guarded as one

Mints a standing delegation credential for the calling user. OAuth cannot express durable delegation: it fuses the grant, the credential, and the session into one refresh token, so an IdP shortening session lifetime shortens the grant. There is no subject parameter — the subject is always the caller's authenticated principal, so cross-subject minting is closed by construction rather than by a check one of seven ports can forget. That is also why this method needs no allowlist and no rate limit while introspect_token has both.

A credential with no verifiable auth_time cannot mint. That single rule is what stops a grant being used to mint another grant and escaping the identity provider permanently, and it makes subprocess and Unix transports fail closed for free, since there is no authenticated principal there at all. A static bearer proves a machine holds a secret, never that a human just authenticated, so it is refused here too. Unlike the introspection rejections, these are deliberately actionable: they are always about the caller themselves, so naming the reason leaks nothing, and it is the only way a console learns to re-prompt.

IssuedGrant carries token, expires_at, and grant_id. The token format is the worker's entirely — a sealed envelope, a database row, or a credential brokered from the IdP are equally valid and equally invisible here. It is never parsed and never logged.

Error kinds

These were an HTTP route whose callers classified definitive-versus-transient on the status code (404 vs 503). As protocol methods every handler exception surfaces the same way, so error_kind carries the whole distinction and is load-bearing rather than decorative.

A caller that negative-caches a transient failure locks out valid users; one that retries a definitive rejection hammers the worker.

Sticky sessions bind handle-bearing process-local state—such as an open cursor, model, or file—to the worker that created it. Both peers opt in; the state stays in process memory and only an AEAD-sealed token crosses the wire.

Teardown is an idempotent DELETE {prefix}/__session__. Lost sessions surface in-band with error_kind = "session_lost"; frameworks do not retry them transparently.