[markstur]

Pull Request

Issue

Fixes #1635

Description

Adds responses endpoint consistent with the OpenAI API and our chat/completions endpoint.
This brings the updated way of interacting which can include server-side storing of context and server-side tools.
The server-side storage is very basic in-memory with time-to-live setting. Storage that lasts across restarts and is more management options could be added later.

Testing

• Tests added to the respective file if code was changed
• New code has 100% coverage if code was added
• Ensure existing tests and github automation passes (a maintainer will kick off the github automation when the rest of the PR is populated)

Attribution

• AI coding assistants used

---

Adding a new component, requirement, sampling strategy, or tool?

If your PR adds or modifies one of the types below, check the matching box. A checklist of type-specific review items will be posted as a comment.

• Component
• Requirement
• Sampling Strategy
• Tool

NOTE: Please ensure you have an issue that has been acknowledged by a core contributor and routed you to open a pull request against this repository. Otherwise, please open an issue before continuing with this pull request.

[markstur]

marking as draft because some fixes are coming

[planetf1]

I ran this against the real OpenAI SDK (openai 2.24.0, with TestClient(app) as the SDK's http_client, so no network needed). The plain non-streaming text path works well, and /v1/chat/completions is unaffected. Beyond that, a few places in the request and stream formats differ from the Responses API, and most of #1635's SDK criteria fail as a result.

Blocking: SDK tool shapes, structured output via text.format, the stream event sequence, streamed turns not being stored, stream not reaching serve(), and instructions carrying over with previous_response_id. Should fix: reasoning-token usage, the server-side tools example, and SDK-level integration tests. The rest are non-blocking suggestions and small nits, all inline.

[planetf1]

The OpenAI SDK sends Responses-style tools, but ResponseTool only accepts the Chat Completions shape. Reproduced with openai 2.24.0, using OpenAI(http_client=TestClient(app)) and a stub serve():

• Tools. The SDK's function tool is flat: {"type": "function", "name", "parameters", "strict"} (openai/types/responses/function_tool_param.py). Only the nested function: {...} form is modelled here, so name and parameters are dropped and serve() receives [{"type": "function"}]. With the server-side tools example, _resolve_tools then hits tool_def["function"]["name"], raises KeyError, and the client gets a 500.
• tool_choice. The SDK form is {"type": "function", "name": ...}. It's passed through unchanged, and the example server reads tool_choice["function"]["name"], so it's ignored.
• Tool results. ResponseInputItem requires role, so a function_call_output item gets a 400. An SDK client can't send a tool result back.

This is the "SDK works without modifications" and "function tool calling works end-to-end" criteria in #1635. Changing the wire shape after people have built against it would be a breaking change. Could we accept the SDK shapes at the edge and normalize them into what serve() gets today? The nested form could stay as an alias so the current examples keep working.

[planetf1]

Fixed. Flat SDK tools and tool_choice are normalised to the nested form, and function_call_output is accepted. Checked with the SDK at e02ab99.

[planetf1]

The SDK sends structured output as text={"format": {"type": "json_schema", "name", "schema", "strict"}}, flat, and Responses.create() has no format argument. Reproduced with openai 2.24.0:

• client_responses_format.py:65 and README.md:383 fail with TypeError: Responses.create() got an unexpected keyword argument 'format'.
• A real text.format request is accepted but ignored: serve() gets format=None, and text ends up in model_options as a raw key.

Could this read text.format in the flat shape (keeping top-level format as an alias if you like) and keep text out of model_options?

[planetf1]

Fixed. text.format reaches serve() as the model and no longer leaks into model_options. Top-level format stays as an alias.

[planetf1]

client.responses.stream() fails with IndexError: list index out of range on the first text delta. That's the call the streaming client example makes (client_responses_streaming.py:30). Reproduced with openai 2.24.0.

The SDK's stream helper builds its snapshot from response.output_item.added and response.content_part.added, then indexes snapshot.output[event.output_index] on each output_text.delta (openai/lib/streaming/responses/_responses.py:330-354). This generator never sends those two events, so the list is still empty when the first delta arrives.

The tool-call events have a related problem. The function_call_arguments.delta and .done payloads (L296-305) have no type or sequence_number. The SDK reads the data body, not the event: line, so they decode as ResponseAudioDeltaEvent with type=None and never match response.function_call_arguments.done. response.failed (L329) is the same. The stream tests check the event: line, which is why they pass.

The fix is the SDK's per-item sequence: output_item.added, content_part.added, the deltas, output_text.done, content_part.done, output_item.done, plus an output_item.added/.done pair around each function call, with type and sequence_number on every event. When there's no text, skip the message item, otherwise it shares output_index 0 with the first function call.

[planetf1]

Fixed. responses.stream() now completes, and the function-call events carry type and sequence_number. Checked with the SDK.

[planetf1]

On the streaming path nothing is written to _response_store. The if request.stream: branch returns before the store write at L542, and stream_response_chunks only uses store to set expires_at.

So a streamed response says it's retained until expires_at, but GET /v1/responses/{id} returns 404, and so does a follow-up turn using previous_response_id. Reproduced with openai 2.24.0: after a streamed turn the entry isn't in the store, and the chained create() raises NotFoundError.

Agent clients usually stream and chain with previous_response_id, so this breaks multi-turn for them. Could the generator store the response, with history built from accumulated_text, just before it emits response.completed? Passing in a small store callback from app.py would keep the store logic in one place.

[planetf1]

Fixed. Streamed turns are stored, so retrieve and previous_response_id chaining work after a stream. Checked with the SDK.

[planetf1]

stream is excluded here and never mapped to ModelOption.STREAM, whereas the chat path maps it (L208). So serve() can't tell the client asked to stream, and the "one per streamed token" behaviour in the stream_response_chunks docstring (streaming.py:197) never happens.

With the streaming example server, ModelOption.STREAM is missing, so it takes the await_result=True branch (m_serve_example_streaming.py:26-41). The whole generation finishes, and then the client gets it as a single delta. Reproduced with openai 2.24.0: stream=True reached serve() with model_options={'temperature': 1.0}.

Mapping it the way the chat path does should be enough, as the generator already handles an uncomputed thunk through astream().

[planetf1]

Fixed. stream=True now reaches serve() as ModelOption.STREAM.

[planetf1]

Union looks unused here.

[planetf1]

Fixed.

[planetf1]

The new snippets here and at L134 use port 8000, but the CLI default is 8080 (commands.py:20). The 8000 at L76 predates this PR.

[planetf1]

Fixed. The snippets now use 8080.

[planetf1]

This file tests cli.serve.app, so it probably belongs in test/cli/ with the other serve tests.

[planetf1]

Fixed. The file is now in test/cli/.

[planetf1]

This mints a fresh message id, so it won't match the item_id the stream used (streaming.py:253). Reusing one id would let a client tie the deltas to the final item.

[planetf1]

Fixed. The streamed item id and the delta item_id now match.

[planetf1]

--response-ttl has no minimum, so 0 or a negative value is accepted.

[planetf1]

Fixed. --response-ttl now has min=1.

[markstur]

Existing feedback has been addressed, but I'm looking into some additional issues after more review.

[markstur]

After addressing the review feedback above, ran a fresh self-review of the full diff and fixed everything it turned up — one commit per fix, grouped by theme below.

Tool-call round trips

• The SDK's store=False pattern (input=response.output + [function_call_output]) was rejected with a 422. The input model now accepts replayed function_call items and output_text content blocks, and ChatMessage gained the tool_calls field (pydantic was silently dropping it before). Parallel calls merge into a single assistant message, so every tool result directly follows the turn that declared it.
• A chained turn after a tool-call response replayed an unpaired tool message (no tool_call_id, no assistant tool_calls), which OpenAI-style providers reject with a 400. The stored history now carries both sides of the pairing through the replay.
• Malformed tool-turn items (missing call_id / name) are rejected at the edge instead of becoming wire messages with None fields; tool_calls is typed and validated; a missing arguments defaults to "{}".

Store / session behavior

• previous_response_id now enforces the TTL the same way GET /v1/responses/{id} does (one shared expiry helper), so the two read paths can't disagree about the same id.
• Streamed turns now store the post-processed output.value as the assistant turn in history, matching the non-streaming path; the stream envelope still shows exactly what was streamed.
• Multimodal history content is stored as plain dicts instead of pydantic objects, so the history actually matches its documented type and stays JSON-serializable.

Edge validation & examples

• Unimplemented standard OpenAI request params (user, reasoning, truncation, metadata, service_tier, and friends) are dropped at the edge instead of leaking into model_options, where strict providers reject them with opaque errors.
• Example clients connect to localhost instead of 0.0.0.0 (a bind address, which fails as a connect target on Windows).

New SDK-level integration tests cover the store=False replay (single and parallel calls) and the chained-turn pairing.

[planetf1]

Approving. I rechecked every point from my earlier review against e02ab99 with the OpenAI SDK, and all are fixed (details in the thread replies).

Nit, non-blocking: a message with content: [] or content: null still returns 200 and serve() receives a None user turn. A 400 would match how input_file is handled, but fine as a follow-up.