Skip to content

feat(openai): accept context state with images in Decisions evaluations - #1147

Merged
mikehostetler merged 2 commits into
agentjido:mainfrom
mhsdef:feat/decisions-context-input
Oct 24, 2026
Merged

mikehostetler merged 2 commits into
agentjido:mainfrom
mhsdef:feat/decisions-context-input

Conversation

@mhsdef

@mhsdef mhsdef commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

Description

ReqLLM.evaluate/4 sends OpenAI Decisions only text or JSON state, so a caller can't reach the endpoint's input_image parts, which the Decisions guide uses to check an image for visible damage. This adds ReqLLM.Context as a state type for OpenAI Decisions:

  • The context compiles to the Decisions input message array: system and user text become input_text and assistant text becomes output_text. Images in user messages become input_image, either inline from ContentPart.image/2 (as a base64 data URL) or from ContentPart.image_url/1.
  • These fail with ReqLLM.Error.Invalid.Parameter before HTTP: tool messages, :file parts, images outside user messages, and empty messages or contexts. The Decisions API rejects all of them. It doesn't accept files, so a PDF has to be rendered to page images first.
  • Other evaluation providers (TypeSafe, OpenRouter, Azure) reject a context state before HTTP, since their drivers JSON-encode the state.
  • Text, map and list state are unchanged.
context =
  ReqLLM.Context.new([
    ReqLLM.Context.user([
      ReqLLM.Message.ContentPart.text("Inspect the product in this photo."),
      ReqLLM.Message.ContentPart.image(File.read!("product.png"), "image/png")
    ])
  ])

ReqLLM.evaluate("openai:gpt-6-luna", context, %{
  visible_damage: %{type: :boolean, instructions: "Does the product have visible damage?"}
})

Type of Change

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature causing existing functionality to change)
  • Documentation update

Breaking Changes

None.

Testing

  • Tests pass (mix test): 5408 passed, 11 skipped, 240 excluded
  • Quality checks pass (mix quality)

New tests:

  • test/provider/openai/decisions_api_unit_test.exs: a context with system, user (text, inline image, image URL) and assistant messages compiles to the expected input array; tool messages, empty contexts and messages, system images and file parts are rejected.
  • test/req_llm/evaluation_test.exs: a context state with an image reaches the wire as input messages through ReqLLM.evaluate/4; a context state for typesafe:jev-latest is rejected before HTTP.

The existing Decisions tests are Req.Test stubs with no recorded fixtures, so the new ones follow that pattern. Against the live API, one synthetic invoice page as a PNG, through ReqLLM.evaluate("openai:gpt-6-luna", context, questions):

document_type: %{"choice" => "invoice", "confidence" => 1.0, "type" => "choice"}
is_invoice: %{"probability" => 1.0, "type" => "boolean"}
total_over_5000: %{"probability" => 1.0, "type" => "boolean"}
input_tokens: 1385

Checklist

  • My code follows the project's style guidelines
  • I have updated the documentation accordingly (ReqLLM.Evaluation docs, README, guides/openai.md)
  • I have added tests that prove my fix/feature works
  • All new and existing tests pass
  • My commits follow conventional commit format
  • I have NOT edited CHANGELOG.md (it is auto-generated by git_ops)

Related Issues

None.


View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

ReqLLM.evaluate/4 sent OpenAI Decisions only text or JSON state, so the
endpoint's input_image parts were unreachable. A ReqLLM.Context state now
compiles to the Decisions input message array: system, user, and
assistant text, and user-message images as base64 data URLs or URLs.
Tool messages, file parts, and images outside user messages fail before
HTTP, and evaluators other than OpenAI Decisions reject a context state.
@mikehostetler mikehostetler added the needs_work Changes requested before merge label Oct 24, 2026

@mikehostetler mikehostetler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking: compile_message/1 ignores Message.tool_calls when the same assistant message has non-empty text. A Context.assistant message with text and tool_calls compiles successfully, but the request contains only output_text. The evaluator then receives incomplete conversation evidence. The Decisions API supports ordered function or custom tool calls and their outputs: https://developers.openai.com/api/docs/guides/decisions#evaluate-a-conversation. Please compile these items, or reject every message with tool_calls before HTTP, and add a regression test.

@mikehostetler mikehostetler left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The blocking issue is fixed in cb50150. Decisions context compilation now rejects non-empty tool_calls before HTTP, so it cannot silently remove conversation evidence. The focused tests, local quality checks, and all GitHub checks pass.

@mikehostetler mikehostetler added ready_to_merge and removed needs_work Changes requested before merge labels Oct 24, 2026
@mikehostetler
mikehostetler merged commit 18af036 into agentjido:main Oct 24, 2026
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants