Repository navigation
feat(openai): accept context state with images in Decisions evaluations - #1147
Conversation
ReqLLM.evaluate/4 sent OpenAI Decisions only text or JSON state, so the endpoint's input_image parts were unreachable. A ReqLLM.Context state now compiles to the Decisions input message array: system, user, and assistant text, and user-message images as base64 data URLs or URLs. Tool messages, file parts, and images outside user messages fail before HTTP, and evaluators other than OpenAI Decisions reject a context state.
mikehostetler
left a comment
There was a problem hiding this comment.
Blocking: compile_message/1 ignores Message.tool_calls when the same assistant message has non-empty text. A Context.assistant message with text and tool_calls compiles successfully, but the request contains only output_text. The evaluator then receives incomplete conversation evidence. The Decisions API supports ordered function or custom tool calls and their outputs: https://developers.openai.com/api/docs/guides/decisions#evaluate-a-conversation. Please compile these items, or reject every message with tool_calls before HTTP, and add a regression test.
mikehostetler
left a comment
There was a problem hiding this comment.
The blocking issue is fixed in cb50150. Decisions context compilation now rejects non-empty tool_calls before HTTP, so it cannot silently remove conversation evidence. The focused tests, local quality checks, and all GitHub checks pass.
Description
ReqLLM.evaluate/4sends OpenAI Decisions only text or JSON state, so a caller can't reach the endpoint'sinput_imageparts, which the Decisions guide uses to check an image for visible damage. This addsReqLLM.Contextas a state type for OpenAI Decisions:inputmessage array: system and user text becomeinput_textand assistant text becomesoutput_text. Images in user messages becomeinput_image, either inline fromContentPart.image/2(as a base64 data URL) or fromContentPart.image_url/1.ReqLLM.Error.Invalid.Parameterbefore HTTP: tool messages,:fileparts, images outside user messages, and empty messages or contexts. The Decisions API rejects all of them. It doesn't accept files, so a PDF has to be rendered to page images first.Type of Change
Breaking Changes
None.
Testing
mix test): 5408 passed, 11 skipped, 240 excludedmix quality)New tests:
test/provider/openai/decisions_api_unit_test.exs: a context with system, user (text, inline image, image URL) and assistant messages compiles to the expectedinputarray; tool messages, empty contexts and messages, system images and file parts are rejected.test/req_llm/evaluation_test.exs: a context state with an image reaches the wire asinputmessages throughReqLLM.evaluate/4; a context state fortypesafe:jev-latestis rejected before HTTP.The existing Decisions tests are
Req.Teststubs with no recorded fixtures, so the new ones follow that pattern. Against the live API, one synthetic invoice page as a PNG, throughReqLLM.evaluate("openai:gpt-6-luna", context, questions):Checklist
ReqLLM.Evaluationdocs, README,guides/openai.md)CHANGELOG.md(it is auto-generated by git_ops)Related Issues
None.
Need help on this PR? Tag
@codesmith-botwith what you need.