Skip to content

fix: clarify the default HITL modification feedback so the LLM uses the modified tool result - #13384

Merged
sjrl merged 1 commit into
mainfrom
fix/hitl-modification-feedback-wording
Oct 22, 2026
Merged

sjrl merged 1 commit into
mainfrom
fix/hitl-modification-feedback-wording

Conversation

@sjrl

@sjrl sjrl commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Related Issues

Proposed Changes:

  • Reword the default modify_template of BlockingConfirmationStrategy to tell the LLM the user wants the modified result used and not to call the tool again with the original arguments.
  • Update the matching tests, docs default and add a release note.

How did you test it?

Unit tests, plus the live script below on main vs this branch with gpt-5-mini: 2/5 runs repeated the tool call on main, 0/5 on this branch.

Notes for the reviewer

With the old wording the model often treated the modified result as a tool failure ("I tried to fetch Berlin, but the weather tool keeps returning Munich") and called the tool again.

Reproduction script (needs OPENAI_API_KEY)
from typing import Annotated

from haystack.components.agents import Agent
from haystack.components.generators.chat import OpenAIResponsesChatGenerator
from haystack.dataclasses import ChatMessage
from haystack.hooks.human_in_the_loop import (
    AlwaysAskPolicy,
    BlockingConfirmationStrategy,
    ConfirmationHook,
    ConfirmationUIResult,
)
from haystack.tools import tool

calls = []


@tool
def get_weather(city: Annotated[str, "The city to get the weather for"]) -> str:
    """Get the current weather for one city."""
    calls.append(city)
    return f"Sunny, 21C in {city}"


class ChangeToMunichUI:
    """Simulates a user who changes every Berlin lookup to Munich."""

    def get_user_confirmation(self, tool_name, tool_description, tool_params):
        if tool_params.get("city") == "Berlin":
            return ConfirmationUIResult(action="modify", new_tool_params={"city": "Munich"})
        return ConfirmationUIResult(action="confirm")


agent = Agent(
    chat_generator=OpenAIResponsesChatGenerator(model="gpt-5-mini"),
    tools=[get_weather],
    hooks={
        "before_tool": [
            ConfirmationHook(
                confirmation_strategies={
                    "get_weather": BlockingConfirmationStrategy(
                        confirmation_policy=AlwaysAskPolicy(), confirmation_ui=ChangeToMunichUI()
                    )
                }
            )
        ]
    },
    max_agent_steps=5,
)

runs = 5
repeated = 0
for _ in range(runs):
    calls.clear()
    result = agent.run(messages=[ChatMessage.from_user("What's the weather in Berlin right now?")])
    # Every tool call reaches the UI with the arguments the model chose, so more than one call means the model
    # asked for the weather again after the user's modification.
    repeated += len(calls) > 1
    print(f"tool calls: {calls} | {result['last_message'].text[:80]!r}")

print(f"\n{repeated}/{runs} runs called the tool again after the modification")

Checklist

  • I have read the contributors guidelines and the code of conduct.
  • I have updated the related issue with new insights and changes.
  • I have added unit tests and updated the docstrings.
  • I've used one of the conventional commit types for my PR title.
  • I have documented my code.
  • I have added a release note file, following the contributors guidelines.
  • I have run pre-commit hooks and fixed any issue.

@vercel

vercel Bot commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
haystack-docs Ready Ready Preview Oct 22, 2026 6:25am UTC

Request Review

@github-actions github-actions Bot added topic:tests type:documentation Improvements on the docs labels Oct 22, 2026
@sjrl
sjrl marked this pull request as ready for review October 22, 2026 06:24
@sjrl
sjrl requested a review from a team as a code owner October 22, 2026 06:24
@sjrl
sjrl requested review from anakin87 and removed request for a team October 22, 2026 06:24
@sjrl sjrl self-assigned this Oct 22, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  haystack/hooks/human_in_the_loop
  strategies.py
Project Total  

This report was generated by python-coverage-comment-action

@sjrl
sjrl merged commit 6f10dd7 into main Oct 22, 2026
36 checks passed
@sjrl
sjrl deleted the fix/hitl-modification-feedback-wording branch October 22, 2026 08:39

This branch was successfully deployed

1 active deployment
Preview — 2dde3a7e Deployed Oct 22, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

topic:tests type:documentation Improvements on the docs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants