Repository navigation
[bot] Merge master/abc7a060 into rel/dev - #1876
Merged
Merged
Conversation
Unless GOODDATA_EVAL_JOIN_GENAI_TRACE is set to an off value, every chat request of an eval item carries W3C baggage naming the item's trace and root span (langfuse_trace_id, langfuse_experiment_item_root_observation_id) and its experiment (langfuse_experiment_id/_name/_dataset_id/_item_id). A gen-ai that accepts caller baggage opens its turn under that root and reports the trace id on response_started. Both ids are derived from the conversation id, so the sending thread and the deferred scorer agree without passing state between threads. For a joined conversation, the deferred task reads the trace by id (/v2/observations?traceId=), waiting until every joined turn is ingested, then exports gd-eval's root into that trace in the sdk-experiment environment, spanning the whole run, and writes each score once on it. Langfuse then reports the item's real cost and latency, and the score rows are halved. The switch is on when unset, so evals get the joined layout by default. Conversations that did not join, and every run with the switch off, keep today's path: poll by session, root in its own trace, scores on both. A joined item reports its slowest turn as latency, each turn timed from its own rows, which is what an unjoined item reports. Cost stays the sum over the trace, so a multi-turn item now carries every turn's spend. A run that joined but gets no scores (judge error, ungraded) still has its root exported and its join released, so gen-ai's turns never sit in sdk-experiment under a missing parent. Unjoined ungraded runs are still not read. The model version behind the experiment name is read once per workspace; resolving it is on the item's clock when the switch is on. The item scope is set before the K runs in all twelve agentic kinds. anomaly_detection and what_if now pass conversation_id to observe, so an unlinked root there now carries session.id. jira: trivial risk: low
parse_sse_lines set ChatError.reason only for a turn that ended without an answer. Plain and transient errors carry gen-ai's reason too, such as the data-obfuscation codes, whose detail is user-facing text. A caller that needs the code reads exc.reason instead of re-parsing the stream, so tavern's obfuscation client no longer has to copy send_message. jira: trivial risk: low
Langfuse Cloud rate-limits public API calls per organization in fixed one-minute windows, and score writes share the bucket with observation and score reads (30/min on Hobby, 1,000/min on Pro). A 429 carries a Retry-After counting down to the window's reset, often 30-60 s. create_score capped every retry wait at 5 s, so both retries landed in the same exhausted window and the score was dropped. On a Hobby project a 60-request burst lost 43 scores this way. A 429 now waits up to the 60 s window; 5xx retries keep the 5 s cap. jira: trivial risk: low
Each eval item wrote every score with its own POST /api/public/scores, twice when the item was not joined to gen-ai's trace. Those requests share Langfuse's public-API rate-limit bucket with the observation and score reads, so a nightly run spent most of its budget on score writes. submit_trace_scoring now collects every score_safe call of the deferred scoring step and sends them as POST /api/public/scores arrays of up to 100. Score ids are fixed before the first attempt, so a retried batch cannot write a score twice. Langfuse validates array entries one by one and answers 207 when some are rejected; that is logged, not retried. Callers outside submit_trace_scoring (the single-shot sink, any client without create_scores) still write each score as it comes. Under a sustained 429 one item now waits at most 2 x 60 s, where per-score writes with the 60 s Retry-After cap could wait 36 minutes, past tavern's 420 s test timeout. A cancelled drain sends what was collected before the interrupt once, without waiting out a retry. jira: trivial risk: low
Langfuse groups API traffic by User-Agent, so every gooddata-eval call showed up as python-httpx/0.28.1, indistinguishable from other callers. Its exported root spans carried only service.name, so a trace did not say which build or CI run produced it. Every client from make_http_client now sends gooddata-eval/<version> (<ci|local>; run=<GITHUB_RUN_ID>; sha=<12 chars of GITHUB_SHA>), with "local" for any value unset or unsafe in a header. The OTLP resource adds service.version and, when set, github.run_id, github.workflow, github.ref_name and github.event_name. jira: trivial risk: low
A score batch that still failed after its retries raised out of send_scores, so every later batch of the item was dropped and the failure count in the log was wrong. The failed batch is now logged with its size and the next batch is sent anyway. The dataset lookup docstring now says it also runs on the item's own thread when the gen-ai trace join is on. _fetch_joined_trace documents that a retried turn can satisfy the ingested-turn count early. AGENTS.md gains GOODDATA_EVAL_JOIN_GENAI_TRACE, item_scope and the orphaned-turn gotcha. Tests cover the failure paths. jira: trivial risk: low
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## rel/dev #1876 +/- ##
===========================================
+ Coverage 84.69% 84.91% +0.22%
===========================================
Files 335 336 +1
Lines 23745 24072 +327
===========================================
+ Hits 20110 20441 +331
+ Misses 3635 3631 -4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🚀 Automated PR to perform merge from master into rel/dev with changes up to abc7a06 (created by https://github.com/gooddata/gooddata-python-sdk/actions/runs/38431652618).