Issues

• Async TurnFaithfulnessMetric assigns window scores in completion order

  confident-ai/deepeval#3480 · Filter by author kausthubhk opened on Oct 18, 2026

• Arbitrary local file read via [DEEPEVAL:IMAGE:...] placeholder in evaluated model output

  confident-ai/deepeval#3475 · Filter by author carfeii opened on Oct 17, 2026

• add_test_cases_from_json_file raises TypeError when tools_called/expected_tools is null

  confident-ai/deepeval#3469 · Filter by author nancy6547 opened on Oct 16, 2026

• [doc] Tool Correctness should not be marked as "Referenceless"

  confident-ai/deepeval#3461 · Filter by author andreabosisio opened on Oct 15, 2026

• Semantics question: should aggregate pass rate be breakable down by the tags/metadata already on LLMTestCase?

  confident-ai/deepeval#3452 · Filter by author Shild-0040 opened on Oct 14, 2026
  2

• New metric: deterministic metadata-filter correctness for retrieval (FilterCorrectnessMetric)

  confident-ai/deepeval#3445 · Filter by author Faizan21-Analyst opened on Oct 12, 2026
  2

• GrokModel.generate/a_generate pass an OpenAI-style list-of-dicts to xai_sdk.chat.user(), so every call crashes with a protobuf TypeError before any network I/O

  confident-ai/deepeval#3436 · Filter by author BlueX888 opened on Oct 11, 2026
  1

• [Feature] Community metric proposal: CalibrationMetric (ECE + Brier) for aggregate confidence trustworthiness

  confident-ai/deepeval#3433 · Filter by author ghubuser-dnt opened on Oct 10, 2026
  5

• LiteLLMModel sends temperature=0.0 by default, causing o-series and Claude Opus 5.5 to fail before making requests

  confident-ai/deepeval#3421 · Filter by author KardeniaPoyu opened on Oct 6, 2026
  1

• AnthropicModel: Claude Sonnet 5.5 not in model registry (temperature 400, no cost, no structured outputs)

  confident-ai/deepeval#3412 · Filter by author VanshBhardwaj24 opened on Oct 4, 2026

• ARC, GSM8K, TruthfulQA, BoolQ and HumanEval fail to load on huggingface_hub 1.16+: legacy dataset ids raise HfUriError

  confident-ai/deepeval#3408 · Filter by author chrikrah opened on Oct 3, 2026

• RAGAS metrics default to gpt-3.5-turbo (shuts down Oct 23), and the TypeScript Grok default is already retired

  confident-ai/deepeval#3404 · Filter by author 23821 opened on Oct 2, 2026

• score_qag_verdicts: remaining conversational call sites still shrink the denominator (follow-up to #3347)

  confident-ai/deepeval#3401 · Filter by author kittimzhe opened on Oct 1, 2026
  2

• Observed async generator resumes under another generator’s span

  confident-ai/deepeval#3398 · Filter by author zmuxuny opened on Sep 30, 2026

• generate_with_schema silently drops the schema when a provider raises TypeError internally

  confident-ai/deepeval#3396 · Filter by author sclfcz opened on Sep 30, 2026

• _send_tool_result: a circular tool result skips the stringify fallback and raises ValueError

  confident-ai/deepeval#3394 · Filter by author sclfcz opened on Sep 30, 2026
  1

• TS verdict metrics credit out-of-vocabulary verdicts as passing (prompt-alignment, role-violation)

  confident-ai/deepeval#3391 · Filter by author kittimzhe opened on Sep 29, 2026

• OpenAI stream=True calls finish their LLM span before stream consumption

  confident-ai/deepeval#3388 · Filter by author dafyy321-pixel opened on Sep 28, 2026

• set-openai changes the active provider before rejecting a missing model

  confident-ai/deepeval#3386 · Filter by author dafyy321-pixel opened on Sep 28, 2026

• evaluate() uses an unset hyperparameters value for cache lookups and writes

  confident-ai/deepeval#3384 · Filter by author dafyy321-pixel opened on Sep 28, 2026

• Async evaluate() / evals_iterator() time out any batch longer than ~207s: gather timeout ignores max_concurrent queueing

  confident-ai/deepeval#3380 · Filter by author hardness1020 opened on Sep 27, 2026

• CitationFaithfulnessMetric: fail out-of-range [N] markers deterministically before the judge call

  confident-ai/deepeval#3376 · Filter by author mikekeror-rgb opened on Sep 27, 2026

• test_core is red on main in three places; --maxfail=1 hides the later two

  confident-ai/deepeval#3372 · Filter by author feiiiiii5 opened on Sep 26, 2026
  1

• Add GPT-6 (gpt-6-sol, gpt-6-luna, gpt-6-astra) to OPENAI_MODELS_DATA so metrics get structured outputs

  confident-ai/deepeval#3366 · Filter by author baptiste-pasquier opened on Sep 25, 2026

• Deepeval tracing

  confident-ai/deepeval#3363 · Filter by author VijayRajIITP opened on Sep 25, 2026