Feature request: build an inference request once and reuse it across retries, without private APIs

[lminer]

Every call to InferenceServerClient.infer() (both tritonclient.grpc and tritonclient.grpc.aio) rebuilds the ModelInferRequest protobuf from the InferInput objects, copying the raw tensor bytes into the message each time. For a client that retries failed sends — e.g. waiting out a model reload or a server restart — this has a large memory cost when inputs are big:

1. The InferInput objects must stay alive across the whole retry loop (one full copy of the tensor bytes), because each attempt rebuilds from them.
2. Each attempt's freshly built protobuf is a second copy — and a failed attempt holds it through the full gRPC wait before the next attempt discards and rebuilds it.
3. gRPC's wire serialization makes a third copy during each send.

So a retrying client holds roughly 3× the payload during an attempt and 2× between attempts. In our production system (audio processing; a whole audio file is sent as one FP32 tensor, 0.5–2 GB per request), retry loops against a temporarily unavailable model produce a repeating sawtooth of the full payload size being re-copied and released per attempt, and were the mechanism behind worker OOMs at our container memory limits.

Describe the solution you'd like

A supported way to separate "build the request" from "send the request", so the expensive copy happens once and only the send is retried. For example either of:

# Option A: expose the builder and accept a prebuilt request
request = client.build_infer_request(model_name=..., inputs=..., outputs=..., ...)
del inputs  # bytes now live only in the request
result = await client.infer_prebuilt(request)  # retryable by the caller

# Option B: a request handle with a public send
request = InferRequest(model_name=..., inputs=..., ...)
result = await client.infer(request=request)

Either shape lets callers drop the InferInputs after building and pay only the (unavoidable) wire-serialization copy per attempt: ~2× payload peak during a send and 1× between attempts, instead of 3×/2×.

Describe alternatives you've considered

• What we do today (works, but uses private APIs): build once with tritonclient.grpc._utils._get_inference_request, then send via client._client_stub.ModelInfer(request=request, metadata=client._get_metadata(None)), converting grpc.RpcError with raise_error_grpc — i.e. exactly infer()'s internal path with the build hoisted out of the loop. It measurably removes ~1× payload sustained plus ~1× payload transient per retry, but it depends on four private symbols and needs re-checking on every tritonclient upgrade.
• Building ModelInferRequest from the public generated protos (tritonclient.grpc.service_pb2) and sending through our own GRPCInferenceServiceStub: fully public, but it means reimplementing the client's request construction and channel handling, and drifting from whatever infer() does.
• System/CUDA shared memory: avoids the copies entirely but only applies when client and server share a host, which is not our topology (clients reach Triton over the network through a load balancer).

Additional context

Measured with a 200 Hz RSS sampler and incompressible random FP32 payloads, comparing rebuild-per-attempt (current infer() behavior, inputs held) against build-once (private-API workaround), k = send attempts including wire serialization:

payload	k	rebuild per attempt, peak	build once, peak
454 MB	3	+2598 MB	+2028 MB
908 MB	3	+5003 MB	+3863 MB
1817 MB	3	+9764 MB	+7485 MB
3634 MB	3	+14727 MB	+11086 MB

The saving is ~1× payload sustained per attempt plus ~1× payload transient per retry, and grows linearly with payload size. Happy to contribute a PR if maintainers agree on a preferred API shape.

---