RFC: Make lemonade-sdk/lemonade the SSOT for Lemonade skills

[danielholanda]

Proposal size

Minor feature (a single PR)

Updates

• The new CI tests will use Devlab Dispatch, so that they don't use the capacity of the existing fixed strix halo runners in sjlab
• Alternative option: The Lemonade API Contracts & Downstream Compatibility Testing proposal enable Lemonade Skills to be maintained as part of it's own lemonade-sdk/skills repo.

User story

Multiple Lemonade contributors (including @iswaryaalex, @sreeram-11, and I) worked on creating 3 Lemonade skills that target a general developer audience. Those skills are in a high-visibility repo (amd/skills). Having those skills living in a separate repo is painful, since breaking changes in Lemonade cause the need to make changes in two repos (lemonade + amd/skills).

The proposal here is to "federate" those skills: skills should live exactly where the product repo lives. Moving those 3 skills to lemonade-sdk/lemonade/skills makes lemonade-sdk/lemonade the single source of truth for those skills.

AMD Skills supports this federation process and the update process is simple: all changes now happen directly on lemonade and amd/skills automatically mirrors those using automated nightly workflows.

This significantly simplifies the development and maintenance process.

High-level design

Move lemonade skills from https://github.com/amd/skills/tree/main/skills to https://github.com/lemonade-sdk/lemonade/tree/main/skills.

Breaking changes

None

Maintenance plan

CI tests for all skills already exist and can be seen inside the evals folder. A simple reusable workflow test file will be added to ensure those tests also run on the lemonade repo.

Risks

No known risks.

[jeremyfowers]

@danielholanda re: maintenance plan, can you clarify:

1. Are the CI tests sufficient to guarantee the skills don't drift with respect to breaking changes? Or is some additional vigilance needed?
2. Do the CI tests require any kind of specialized hardware, API keys, etc?

If the CI is self-sustaining then I don't see any issue with moving the skills here.

[danielholanda]

1. CI tests are sufficient to catch major breaking changes and were able to demonstrate that in the past. That said, the agents that run those skills are quite resilient and they may be able to go around minor breaking changes. As a result, we should be able to rely on those tests when changing Lemonade itself, but we shouldn't rely only on tests when updating/improving the skill itself.
2. STX Halo is recommended (not required) for the tests as local-ai-use performs a quick image generation task. An Anthropic API key is also needed (and I may be able to assist the project getting access to one for skills testing only, if an anthropic key is not already available). Tests are usually fast and only take 2-3min each (example).

[jeremyfowers]

STX Halo is recommended

Please update the RFC body to reflect that the new CI tests will use Devlab Dispatch, so that they don't use the capacity of the existing fixed strix halo runners in sjlab.

Tests are usually fast and only take 2-3min each

Some more questions:

• How many tests are there, and what is the total wall clock time of the test suite?
• Can you demonstrate that the tests are 100% deterministic?
• How do you want the tests to be triggered? Is merge queue only (ie, not on every PR push) ok?

Anthropic API key is also needed

You have to provide your own Anthropic key and budget. You also have to guarantee the merge queue will never become blocked because you ran out of budget, or because Claude services are down (which would be nondeterminism). It might be better to migrate your tests off of Anthropic API and onto something like Pi agent that can run on the CI runner itself.

[jeremyfowers]

something like Pi agent

or lemonade launch claude on strix halo!

[danielholanda]

• How many tests are there, and what is the total wall clock time of the test suite?

  How many: Across local-ai-use, local-ai-app-integration and lemonade-router-builder there are 27 routing cases (checks on whether the skills are triggering correctly + 7 behavioral cases with 37 checks in total (each case is an end-to-end test with multiple checks for each of those end-to-end runs).

  Time: Less than 5min for routing total + 7/3.5/5 min for the behavioral tests of each skill

• Can you demonstrate that the tests are 100% deterministic?

  Nope. That would be a lie. Tests run real agents, which are inherently non-deterministic. That said, I can confirm I ran all lemonade tests described above 3x in a row (118 total), and all of them passed.

  That said, even though those tests are all reliably passing now, I would still recommend making those tests advisory instead of blocking.

• How do you want the tests to be triggered? Is merge queue only (ie, not on every PR push) ok?
  I think merge queue would be enough for now while we are using actual cloud tokens and then move it to every PR push once we hook up Lemonade as a backend for the actual skills).

[jeremyfowers]

What sorts of changes would qualify as a "breaking change" that would fail CI and require edits to the skills? I'm asking because I'm concerned that routine Lemonade development will incur a maintenance tax.

[danielholanda]

There are three types of skill tests we care about here:

Structural tests

Check the actual structure of the skill itself (missing links, description length, etc). No changes to code Lemonade would break this.

Routing tests

Checks whether the actual skill is triggered when it should and not triggered when it shouldn't. This only breaks if you change the actual description of Lemonade, not Lemonade itself. So, also safe here.

Behavioral tests

Check the behavior of the skill end-to-end. This is where breaking changes in Lemonade cause the Lemonade skills to break. You can check the "evals.json" files of each of the skills to see how those are structured. Here is an example:

"id": "generate-cat-image",
      "skill_should_trigger": true,
      "prompt": "Learn how to generate images locally, then generate an image of a cat and save it to out.png.",
      "expected_behavior": [
        "Install Lemonade Server if it is not already installed",
        "Download the SD-Turbo model if the model is not already downloaded",
        "Add a 'Local AI Use' block to AGENTS.md"
      ],
      "unexpected_behavior": [
        "Pull TTS or STT models for this image generation task",
        "Reach for a cloud image path instead of local Lemonade"
      ],
      "files_exist": [
        "AGENTS.md",
        "out.png"
      ]

As you can imagine here, in most cases, the skill would only break if Lemonade made a major change, such as changing the pull command or dropping image generation support.

[jeremyfowers]

Would you be open to changing some of these so that breakage is less likely? For example

"Download the SD-Turbo model if the model is not already downloaded",

Could be

"Download the smallest image generation model, if no image generation model is not already downloaded",

and then this will work even if we drop SD-Turbo support for some reason, as long as we still have image generation support at all.

[jeremyfowers]

Have you considered how this interacts with the release cycle? Any breaking change would become effective on the skill on main when the breaking PR merges, but the breaking change won’t hit clients until the release up to 14 days later.

[danielholanda]

Skills are explicitly tracking the latest release. We also have a mechanism when importing the skills to amd/skills where we only update the actual skill when a new version has been released. This leaves us with the following scenario:

Users on AMD skills OR users on a release - Users are pushed towards the latest release and skills work as expected.

Users on main - Skill development can only happen once a release has been published anyways, since the skills point to the latest release. As a result, skills on main should also be stable.

This strategy gives us stability, at the cost of having to wait for features to be added to a release before those can be added to a skill. Let me know what you think of this.

---

Additional thoughts: A real risk is having someone that installed a skill a long time ago and the "Latest" version of lemonade no longer works with the old skill (example: a specific backend or model name dropped). We should consider hardcoding the recommended version of Lemonade to each skill to address this issue.

[danielholanda]

Added the following update to the RFC:
The #3732 proposal enable Lemonade Skills to be maintained as part of it's own lemonade-sdk/skills repo.

I think this is also a reasonable solution here.