RFC: Make Lemonade capabilities server-owned and reusable through MCP

[fl0rianr]

Goal

Make Lemonade's capabilities server-owned and expose them as reusable MCP-ready tools instead of implementing execution and orchestration separately in each client.

GUI3's lemonadeTools.ts https://github.com/lemonade-sdk/lemonade/blob/96a8bd5cd3b5195eba009f147329e1ef65f534b9/src/app/src/tools/lemonadeTools.ts currently contains a significant amount of its own execution, model-selection, and orchestration logic. That creates duplication and means another UI, CLI, agent, or external application would have to rebuild the same Lemonade-specific behavior, this is not the intended behavior.

The desired end state is:

• Lemonade owns model discovery, capability selection, orchestration, and execution logic.
• /mcp exposes these capabilities through stable, reusable tool contracts.
• GUI3 becomes a consumer of those contracts rather than a second implementation.
• The same capabilities are directly reusable by other UIs, CLIs, agents, and applications embedding Lemonade.

This directly supports a core Lemonade product goal: making Lemonade easy to integrate into other applications.

Current work toward that end state is split across #3319, #3346, #3350, #3372, and #3373. These should be reviewed as parts of this common architecture rather than as unrelated MCP features.

Made to make this case clearer but also stated in #3275 etc.

[jeremyfowers]

@fl0rianr would you kindly list the new MCP tools being added, so we fully understand the scope?

@siavashhub @kpoineal please review this proposal at a conceptual level, in accordance with the guidelines in #3421

1. Is this the long-term feature scope we want for Lemonade's MCP server?
2. Does all of this need to be release-gating for GUI3, or can we implement a sub-set, then release GUI3, then add more MCP tools later?

[fl0rianr]

We discussed on the lengthy topic on discord that GUI3 should have it's own HTTP MCP. But the server is insufficient regarding that. Making a bunch of own calls and server calls instead of getting ONE valid tool call answer for that call. As such execution is a server topic.

[jeremyfowers]

We discussed on the lengthy topic on discord that GUI3 should have it's own HTTP MCP. But the server is insufficient regarding that. Making a bunch of own calls and server calls instead of getting ONE valid tool call answer for that call. As such execution is a server topic.

Yes, but that doesn't mean a very large amount of MCP tools need to be brought into the server ASAP as a GUI3 release blocker.

Alternatives include:

• Releasing GUI3 with the lemoandeTools.ts in place, and gradually migrating to MCP
• Dropping the presently unsupported MCP tools from GUI3 MVP and adding them back in post-MVP as they become available in the server.
• Never supporting certain proposed tools in GUI3 (not saying we should do this, but it's an option)

[jeremyfowers]

Made to make this case clearer but also stated in #3375 etc.

@fl0rianr I think you linked the wrong issue, #3375 does not seem relevant.

edit: I think you meant #3275

[fl0rianr]

Made to make this case clearer but also stated in #3375 etc.

@fl0rianr I think you linked the wrong issue, #3375 does not seem relevant.

edit: I think you meant #3275

Ha, this one was a human error, thanks for noting

[kpoineal]

As per Jeremy's concerns - Which tools from GUI3 are you proposing we move to the backend in Lemonade?

I think most of them should be moved, but if you have a specific tool you wanted to move prior to the GUI3 release then we should discuss. My current understanding was that we would release GUI3 and then work on the MCP backend and roll those tools over.

FYI there is already an existing MCP Server with some tools, so there is a little overlap as well.

[fl0rianr]

Here is a detailed version of the short on above which should clarify open questions:

Goal

Make Lemonade capabilities server-owned and reusable across clients, with /mcp exposing them through stable tool contracts.

The goal is not to move Lemonade business logic into MCP. Lemonade should own model discovery, model selection, execution, validation, and orchestration once on the server side. REST and MCP should then be adapters to that same behavior.

Today GUI3 still implements a significant amount of Lemonade-specific behavior in lemonadeTools.ts, including model selection, request construction, media execution, and orchestration. That means another UI, CLI, agent, or application embedding Lemonade would need to rebuild similar logic.

The intended architecture is:

                Lemonade capability / business logic
              discovery, selection, execution, orchestration
                              |
                 +------------+------------+
                 |            |            |
                REST         MCP        other clients
                 |            |
           Lemonade App   MCP clients

This directly supports a core Lemonade product goal: making Lemonade easy to integrate into other applications.

Long-term MCP surface

This is the desired capability surface, not a statement that every item below must block the initial GUI3 release.

Model discovery

Tool | Purpose -- | -- lemonade_list_models | Query models registered/supported by this Lemonade server lemonade_get_model_info | Detailed record for one model lemonade_get_loaded_models | Current runtime residency lemonade_search_models | Search supported remote registries lemonade_get_pull_variants | Inspect pullable variants of a remote repository

What is actually being added

The PRs are not 21 unrelated new MCP features.

• MCP: expose canonical Lemonade capability hub #3319 expands /mcp into the capability hub and exposes the server capabilities that already exist behind Lemonade APIs.

• server: make registered model discovery canonical and queryable #3346 adds the canonical server-side model query/capability contract. It does not add another MCP tool.

• MCP: expose remote model search and pull variant discovery #3350 adds the two genuinely new discovery tools: lemonade_search_models and lemonade_get_pull_variants.

• MCP: select image-edit capable models for edit_image #3373 fixes lemonade_edit_image model selection so editing requires an actual image-edit capable model.

• MCP: add prompt-to-3D orchestration #3372 extends lemonade_generate_3d with server-side prompt -> reference image -> 3D orchestration.

They should therefore be reviewed as pieces of the same architecture rather than as unrelated MCP features.

Model selection

Model-backed tools should follow one reusable server-owned policy:

1. explicit compatible model from the caller

2. already loaded compatible model

3. already downloaded compatible model

4. an optional known default only where that tool explicitly permits an automatic download

Compatibility is capability-based.

For example:

generate_image     -> image
edit_image         -> image + image-edit
transcribe_audio   -> transcription
generate_audio     -> audio-generation
text_to_speech     -> tts
generate_3d        -> model3d
omni               -> omni

An IMAGE model is not automatically capable of image editing.

Likewise, inference tools should not silently perform remote registry searches.

What remains host/UI responsibility

Server-owned behavior does not mean moving conversation behavior into Lemonade.

The host still owns:

• selecting the attachment the user refers to

• "use the latest image" semantics

• conversation history

• file pickers

• approval UX

• progress/cancellation UX

• icons, grouping and rendering

For example, lemonade_edit_image should receive the actual image to edit. The server should not inspect GUI3 conversation history to determine what "the latest image" means.

GUI3 release scope

The long-term architecture above should not imply that every tool is release-gating for GUI3.

GUI3 can migrate capabilities incrementally.

The requirement for a migrated capability is simply:

Once GUI3 consumes that capability through MCP, the server/MCP contract must be sufficient so GUI3 does not need a second implementation of Lemonade-specific execution or model-selection behavior.

Capabilities that GUI3 still implements locally can be migrated later without changing the architecture described here.

In particular, remote model discovery and higher-level orchestration do not need to become artificial GUI3 release blockers merely because they belong in the long-term MCP surface.

The exact GUI3 MVP cut can therefore be decided separately from the long-term server contract.

tools/list as the MCP contract

For migrated capabilities, tools/list should be the runtime source of truth for:

• tool name

• description

• input schema

• MCP annotations

GUI3 should not maintain a second copy of those server contracts.

UI-only metadata remains local.

Safety

Lifecycle/backend tools modify Lemonade state. Their existence in the MCP implementation and whether an arbitrary agent is allowed to invoke them are separate concerns.

Mutating/destructive tools should have accurate MCP annotations and appropriate guards. A server-side configuration/permission mechanism can additionally control whether control-plane tools are exposed.

The important architectural point is that authorization and host approval remain separate from capability implementation.

Intended end state

After this work:

• Lemonade owns its capability behavior once

• REST and MCP reuse that behavior

• /mcp exposes a discoverable Lemonade tool surface

• model selection is capability-based and server-owned

• remote discovery is separate from local discovery and mutation

• GUI3 can progressively remove duplicated Lemonade execution/model-selection logic

• other UIs, CLIs, agents, and applications can reuse the same capabilities

This is the user-facing benefit: Lemonade becomes substantially easier to embed and control as a local AI platform without every client having to reimplement Lemonade-specific behavior.

Related: #3275, #3319, #3346, #3350, #3372, #3373.

[kpoineal]

I'd appreciate it if you could tell in your own words what you're trying to do. I am aligned with serving lemonade tools from the MCP instead of the GUI3 client. Is this discussion that broader work scope, or are you saying we need to implement the tools on MCP prior to GUI3 release?

[fl0rianr]

I'd appreciate it if you could tell in your own words what you're trying to do. I am aligned with serving lemonade tools from the MCP instead of the GUI3 client. Is this discussion that broader work scope, or are you saying we need to implement the tools on MCP prior to GUI3 release?

Since the server side PRs are already there we should try to review and merge those.
In parallel we can also tackle the GUI3 side you can do it or I can to it as well as I can't review my own work anyway. Some of that GUI3 work is depended some is not.

During this process we can align/decide if we finalize the server side and move on to the final GUI3 depended work or we do the depend work on GUI3 and pause before a full scope server solution.

I want to note that originally I only wanted to enhance the tools/list. But like in the intense dev phase recently together with @jeremyfowers where we identified and made clear that e.g. load (and many more) is a server topic and there should not be any "special magic" used on GUI side. E.g. List models is one of these as well which motivated me for a more broader fix. As I was assuming we won't merge GUI3 before that.

[jeremyfowers]

Since the server side PRs are already there we should try to review and merge those.

Yes eventually, but the core idea of the new review philosophy is that we shouldn’t feel rushed to merge code. 2500 LoC across 5 PRs is a great example of something we shouldn’t rush. Especially since I want @siavashhub to review anything MCP-related and I know he has limited time.

I am hoping we can prioritize which MCP features are required for GUI3 MVP, review and merge those, merge GUI3, and then come back to the rest later.

[jeremyfowers]

PS. just to be 100% clear, I have no issue with some tools being defined in lemonadeTools.ts and then migrated to MCP later.

If this entire body of work is an attempt to migrate all tools to MCP prior to GUI3 release, I propose we defer all of it until post-GUI3 release.

The only thing that would block GUI3 release for me on the server side is if GUI3 relies on unreviewed C++ code that went straight into GUI3_merging. There is no blocker with functionality enabled via pure TypeScript.

[fl0rianr]

For the immediate path, I would propose #3319 as the minimum MCP foundation to merge ASAP, together with #3350 as its remaining test/CI issue is resolved. #3350 is a small read-only extension to that foundation and already has conceptual approval from @siavashhub.

I would not make #3346, #3372 and #3373 hard GUI3 release blockers. They improve the same architecture and can land progressively.

So: #3319 + #3350 = near-term merge target, not everything must be finished before GUI3 ships.

[jeremyfowers]

Thanks for the helpful scope reduction @fl0rianr !

@kpoineal what are your thoughts?

[jeremyfowers]

Actually I still want to push back on:

For the immediate path, I would propose #3319 as the minimum MCP foundation to merge ASAP, together with #3350 as its remaining test/CI issue is resolved.

There are 3 categories of work:

1. Aspirational features, that we want to add to the roadmap
2. Roadmapped features, that are are committed to adding eventually
3. Immediate needs, for example blocking bugs or roadmap commitments that apply to the next 1-2 weekly releases

I know that we discussed MCP features aspirationally. But if our goal is to release GUI3 ASAP, what puts 3319 and 3350 into category 3? Why should we spend valuable bandwidth on them now (and presumably in integrated them into GUI3), instead of deferring them and spending our limited time focused on releasing GUI3 with lemonadeTools.ts untouched?

Let's put aside sunk costs and be disciplined about what we commit to.

[fl0rianr]

If we're fine shipping lemonadeTools.ts untouched, that shifts the focus quite a bit.

The two server-side PRs already exist, while the two corresponding GUI3 pieces still don't. I was expecting others to pick those up, which is why I opened the issues as requested:

• GUI3: use the official MCP TypeScript SDK for HTTP MCP #3270 for direct HTTP MCP support in GUI3
• GUI3: support local stdio MCP servers through Tauri #3271 for local stdio MCP support through Tauri

So from my perspective, those are now the missing pieces rather than more work around lemonadeTools.ts.

[siavashhub]

I think the adapter model with the core owning the capability logic and /mcp acting as a thin adapter over the same implementations as our APIs is the right shape for the gateway.

The guardrail I'd keep is the one I mentioned earlier, /mcp tools should exposing existing capabilities rather than becoming a new agent-style orchestration layer. Composing our own primitives into a single media capability is fine, but multi-step tool loops or the gateway itself hosting external tools is where I'd draw the line which we already discussed.

I consider the mutating-tool exposure settled by the decision in #3319.

I also agree with @jeremyfowers on not blocking and shipping GUI3 with its current implementation then migrate incrementally starting with the read-only foundation in #3350.

[SlawomirNowaczyk]

I fully agree with @siavashhub: the key is to keep the /mcp very thin.

If we do that, the discussion of whether it's required for GUI3 or not becomes somewhat moot: once /mcp becomes both easy and useful, it's clearly beneficial to add it quickly, and start using it immediately.

In my opinion, much of this discussion is muddled by the aspirational vision of /mcp... and that we should delay.
Instead, let's figure out a good starting point, and implement that.

[jeremyfowers]

multi-step tool loops

FWIW we do support multi-step tool loops in the codebase for Omni Collections. But perhaps you’re right this should be out of scope for MCP: for anyone using the MCP server, its implied that they are hooked into an agent already, and that agent can do its own tool loop. A good example might be Trellis 3D: trellis requires an image as input; our MCP can state that and state that using our image gen MCP tool is a great way to obtain an image. No need for us to call image gen internally to lemonade in this instance.

I think the adapter model with the core owning the capability logic and /mcp acting as a thin adapter over the same implementations as our APIs is the right shape for the gateway.

Architecturally, is the following the right concept?

1. We need a struct/class for endpoint definition, similar to how we have one for new backends. This class should ensure proper construction, including…
2. Each endpoint class instantiation should include the required information for exposure as an MCP tool (and a flag for whether it should be enabled as such), for example the tool’s description string should be a member of this class.
3. The MCP server code itself ought to just scan all the registered endpoint classes and expose anything with a tool description and the enabled=true flag.

[PCAssistSoftware]

I will need more time to fully read this and all the linked issues / PR's etc. but an initial comment would be that I think we need to prioritize getting GUI3 out of the door first and then looking to do any work towards MCP's. So I would favor the option of "Releasing GUI3 with the lemoandeTools.ts in place, and gradually migrating to MCP" as I think GUI3 beta / release really does need to be the first priority, as seeing all the comments on other websites / Discords about Lemonade it seems a lot of people are excited about the GUI3 release and don't like the current GUI2.

When I have more time this week, I will read through this RFC in more detail and add any additional comments.

I would also like to know what MCP tools are planned to be added?

[siavashhub]

Architecturally, is the following the right concept?

1. We need a struct/class for endpoint definition, similar to how we have one for new backends. This class should ensure proper construction, including…
2. Each endpoint class instantiation should include the required information for exposure as an MCP tool (and a flag for whether it should be enabled as such), for example the tool’s description string should be a member of this class.
3. The MCP server code itself ought to just scan all the registered endpoint classes and expose anything with a tool description and the enabled=true flag.

yes, I think the declarative self-registering direction is right. It matches the pattern used by mainstream MCP SDKs, where metadata is co-located with the handler and the server exposes the registry. It also prevents drift caused by maintaining the JSON catalog in mcp_server.cpp separately from the handlers.

The one thing I'd avoid is a design where every endpoint is a candidate tool, since an MCP tool isn't the same abstraction as a REST endpoint. Good tool catalogs are deliberately developed and kept high level for an agent to reason over rather than being a 1:1 reflection of the API surface. Too many or overly granular tools hurt tool selection and consume context. The mapping may not be 1:1 either, one tool may compose several endpoints, while some endpoints should not be tools at all.

I imagine each descriptor carrying an explicit MCP contract like name, description, input schema and result adapter rather than deriving it from the REST signature. Our MCP contracts already intentionally differ from their REST counterparts (delete_model requires confirm: true, unload_model does not expose the empty-name "unload all" behavior and media tools adapt binary results).

As an option we can also have an exposure tier (off / read-only / mutating) instead of a boolean to keep exposure fail-closed and declare the read-only versus mutating distinction in one place.

Overall, if "scan the registered endpoints and expose the enabled ones" means scanning a curated registry of hand-authored tool contracts, I think it's the right concept.

[jeremyfowers]

I think we can close this out, good discussion all!

#3619 performs the requested refactor, which makes the MCP codebase maintainable as we add many more tools.

The proposed additional tools are uncontroversial, with the exception of the tools that can perform destructive actions. Those should be PR'd in one at a time, and we should experiment with them in real world usage to see if agents go rogue and take unexpected destructive actions.

Any concerns please DM me on Discord!