Repository navigation
Keep cache_control and other extra keys on OpenAI content parts in ChatMessage #13139
Closed as not planned
Description
Since #12446, ChatMessage.from_openai_dict_format turns list content into Haystack content parts and drops extra keys on those parts, such as cache_control. Before, the raw list was sent back unchanged, so prompt caching through OpenRouter or LiteLLM worked by accident. Now it silently stops. OpenAI ignores extra keys on content parts, so sending them back is safe there.
Options
- Reuse
meta["cache_control"](preferred). Movecache_controlfrom a message's parts into the meta key the Anthropic integration already reads, and haveto_openai_dict_formatput it back on the message's last text part. This restores caching through OpenRouter and LiteLLM and letsAnthropicChatGeneratorcache these messages. A breakpoint placed mid-message moves to the end of the message. - Per-part extras. Add an
extradict toTextContent, likeFileContenthas, and send each part's unknown keys back. Exact for any key, but it changes a public dataclass, stops joining text parts in system and assistant messages, and passes unchecked keys to every OpenAI-compatible server again. - Keep the raw list in meta and resend it. Lossless, but it doubles image payloads, goes stale when the message is edited, and skips the validation from fix!: parse list-of-parts content in ChatMessage.from_openai_dict_format #12446.
Open question: do stricter OpenAI-compatible servers, such as vLLM, accept cache_control?
👋 Hello there! This issue will be handled internally and isn't open for external contributions. If you'd like to contribute, please take a look at issues labeled contributions welcome or good first issue. We'd really appreciate it!
Activity
anakin87 commentedon Oct 12, 2026
I investigated the topic a bit
cache_controlis not an OpenAI field (accepted but ignored)ChatMessage.from_openai_dict_formatis a convenience method for users to convert OpenAI dictionaries into HaystackChatMessage, never used in Haystack/Core IntegrationsFor these reasons, I'd be inclined to leave the behavior unchanged (apart from refactoring these conversion functions that got intricate)
@julian-risch @sjrl feel free to add more details. There may be uses of
ChatMessage.from_openai_dict_formatthat I'm overlooking.anakin87 commentedon Oct 12, 2026
Studying this a bit more, I also realized that part of the complexity of
to_openai_dict_format/from_openai_dict_formatcomes from handling "refusal".But I also found that:
OpenAIChatGeneratordoes not use thse helpers and does not preserve this fieldChatMessage.meta["refusal]"So, unless there is a use case I am missing, I think that removing this "refusal" handling could simplify the conversion and making this code easier to understand and maintain in the long run.
sjrl commentedon Oct 13, 2026
@anakin87 I guess a question for me would be should we add the
refusalhandling to theOpenAIChatGenerator? Its possible this is a feature we have been missing out on or ignoring that we should support as part of the OpenAI spec. WDYT?anakin87 commentedon Oct 13, 2026
@sjrl Yes, I think you're right
julian-risch commentedon Oct 14, 2026
Leaving the behavior unchanged except for what you talked about
refusalhandling sounds good to me too!For context, Hayhooks relies on from_openai_dict_format in a couple of places and I believe what worked in the past is that if a user set
cache_controlon content parts and had that in the content list, Hayhooks passed this through withfrom_openai_dict_formatandto_openai_dict_formatand an OpenRouterChatGenerator or LiteLLMChatGenerator could actually receive that and act on it for example by enabling caching for anthropic models.anakin87 commentedon Oct 14, 2026
I'll do the following:
cache_controland other extra keys (conversion methods in Hayhooks are just used in code examples)remove refusal handling from conversion methods for nowOpenAIChatGeneratoranakin87 commentedon Oct 14, 2026
I opened #13255 to track the support for refusals in OpenAI Chat Generators