Add chat streaming for completions#471
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new streaming API to LLMModel/LiteLLMModel that yields incremental text deltas from provider streaming, followed by a single canonical terminal LLMResult containing the completed Message/ToolRequestMessage (keeping existing non-streaming call paths intact).
Changes:
- Introduces
LLMModel.call_stream()and plumbing to request Chat Completions streaming with delta-yielding support. - Refactors Chat Completion parsing into
_parse_chat_completion()and enhances custom tool parser type detection. - Improves stream lifecycle handling (
aclose) and adds/updates tests + README docs for streaming behavior.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/lmi/src/lmi/llms.py | Adds call_stream, stream config validation, stream delta yielding in acompletion_iter, stream closure improvements, and parsing refactor. |
| packages/lmi/src/lmi/cost_tracker.py | Adds TrackedStreamWrapper.aclose() to support contextlib.aclosing on tracked provider streams. |
| packages/lmi/tests/test_llms.py | Adds realistic stream-chunk builders and comprehensive call_stream tests (text + tool call assembly + closure + validation/callback behavior). |
| packages/lmi/tests/test_dispatch.py | Adds coverage ensuring _commit_stream closes the underlying source iterator when the committed stream is closed early. |
| packages/lmi/tests/test_cost_tracking.py | Adds coverage ensuring TrackedStreamWrapper.aclose() forwards to the wrapped stream. |
| packages/lmi/README.md | Fixes a typo and documents call_stream usage/semantics. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| async for item in iterator: | ||
| yield item | ||
| finally: | ||
| if aclose := getattr(iterator, "aclose", None): |
There was a problem hiding this comment.
Maybe worth a docstring change to note when aclose is expected
|
|
||
|
|
||
| def _tool_parser_accepts_text(parser: Callable[..., Any]) -> bool: | ||
| """Return whether a custom tool parser accepts completed text.""" |
There was a problem hiding this comment.
Can you flesh this out? I am not sure what it means for a tool parser to accept completed text vs not. Like maybe provide an example or just write out what's happening - seems like we're checking if the first arg is a string? Also what happens if it's str | ...?
yield incremental completion tokens as they stream in from the provider followed by a
LLMResultcontaining the completedMessageorToolRequestMessage. The former supports streaming UIs while the latter constitutes the canonical completion. existing calls in non-streaming contexts are unchanged.