Skip to content

fix(llm): retarget the deadline on the existing client instead of rebuilding it - #520

Open
FacaHD wants to merge 1 commit into
NVIDIA:mainfrom
FacaHD:fix/async-client-cleanup
Open

fix(llm): retarget the deadline on the existing client instead of rebuilding it#520
FacaHD wants to merge 1 commit into
NVIDIA:mainfrom
FacaHD:fix/async-client-cleanup

Conversation

@FacaHD

@FacaHD FacaHD commented Sep 10, 2026

Copy link
Copy Markdown

Fixes #519

Problem

Under a workflow deadline (SKILLSPECTOR_MAX_WORKFLOW_SECONDS), large scans emitted a stream of:

Task exception was never retrieved
RuntimeError: Event loop is closed

Root cause

Every batch has to be sent with whatever time is left, and _model_for_call() expressed that by building a fresh chat model per call. LangChain caches the underlying httpx client in an lru_cache keyed on (base_url, timeout, socket_options), so a deadline shrinking by a few milliseconds per call is a new cache key every call. A long scan opened one connection pool per LLM request and began evicting the oldest once it passed 128.

Eviction is what crashed. Each analyzer node is synchronous and reaches run_async()asyncio.run(), so it owns its event loop. A pool evicted while a later analyzer is running belongs to an earlier, already-closed loop, and the OpenAI SDK's finaliser reacts to garbage collection with asyncio.get_running_loop().create_task(self.aclose()) — closing the earlier analyzer's sockets from the current analyzer's loop. Nothing awaits that task, so it surfaced only as "Task exception was never retrieved", once per eviction.

Fix

Apply the deadline to the client that already exists instead of baking it into a new one.

  • New _retarget_request_timeout() in llm_analyzer_base.py mutates client.timeout on the live client for ChatOpenAI, ChatAnthropic, and AgentCLIChatModel. Both SDKs read client.timeout when building each request — the same in-place mutation _uses_native_connection_retries already relies on for max_retries.
  • _model_for_call() retargets first and only falls back to constructing a replacement model for transports that keep no mutable deadline, so behaviour is unchanged for them.
  • _StructuredAgentCLIModel now reads transport settings through its owner rather than copying them at construction, so a structured wrapper no longer pins the deadline it happened to be created with. with_structured_output wraps that same model instance.

Dynamic timeouts, the explicit retry loop, the global concurrency limiter, and the one-loop-per-analyzer execution model are all untouched.

Result: one pool per analyzer, one cache key, nothing to evict, and no cleanup that can outlive its loop.

Validation

  • A 200-call reproduction against the openai 3.11 / httpx 2 combination that reported the failure goes from 75 unretrieved Event loop is closed tasks and 220 httpx clients → 0 and 20 (one per analyzer).
  • Regression tests added in tests/nodes/test_llm_analyzer_base.py and tests/unit/test_llm_utils.py.
  • make test (unit): 4015 passed, 14 skipped, 4 xfailed.
  • make lint: clean. make format / ruff format --check: no changes (198 files already formatted).
  • A real scan over an internal skills root produced zero Event loop is closed / Task exception was never retrieved messages.

Note: tests/integration/test_graph.py::test_graph_surfaces_degraded_llm_stage fails in my local environment, but it fails identically on unmodified main (verified by reverting only the two source files) — it is pre-existing and unrelated to this change.

@FacaHD
FacaHD force-pushed the fix/async-client-cleanup branch 2 times, most recently from 9a4258a to 56eab3f Compare September 10, 2026 10:44
…uilding it

A workflow deadline makes every batch carry a slightly different timeout, and
`_model_for_call` expressed that by building a fresh chat model per call.
LangChain caches the underlying httpx client in an `lru_cache` keyed on
`(base_url, timeout, socket_options)`, so a continuously shrinking deadline is a
new cache key every call: a long scan opened one connection pool per LLM request
and started evicting the oldest once it passed 128.

Eviction is what crashed. Each analyzer node is synchronous and reaches
`run_async()` -> `asyncio.run()`, so it owns its event loop. A pool evicted while
a later analyzer is running belongs to an earlier, already-closed loop, and the
OpenAI SDK's finaliser reacts to collection with
`asyncio.get_running_loop().create_task(self.aclose())`. Nothing awaits that
task, so it surfaced only as "Task exception was never retrieved:
RuntimeError: Event loop is closed", once per eviction.

Apply the deadline to the client that already exists instead. Both SDKs read
`client.timeout` when building each request, which is the same in-place mutation
`_uses_native_connection_retries` already relies on for `max_retries`, and
`with_structured_output` wraps that same model instance. Transports without a
mutable deadline still fall back to constructing a replacement, so behaviour is
unchanged for them.

Dynamic timeouts, the explicit retry loop, and the global concurrency limiter are
untouched, as is the one-loop-per-analyzer execution model.

A 200-call reproduction against the openai 3.11 / httpx2 combination that
reported the failure goes from 75 unretrieved `Event loop is closed` tasks and
220 httpx clients to 0 and 20 (one per analyzer).

Signed-off-by: FacaHD <49478213+FacaHD@users.noreply.github.com>
@FacaHD
FacaHD force-pushed the fix/async-client-cleanup branch from 56eab3f to a352111 Compare September 10, 2026 10:48
@FacaHD
FacaHD marked this pull request as ready for review September 10, 2026 11:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Dynamic workflow deadline leaks an httpx pool per LLM call and raises "Event loop is closed"

1 participant