Lemonade API Contracts & Downstream Compatibility Testing

[iswaryaalex]

Proposal size

Minor feature (a single PR)

Updates

No response

User story

Lemonade already has API and integration tests covering much of the lemond server functionality.

As Lemonade is increasingly consumed by downstream applications such as Unsloth, we need to make the existing coverage explicitly protect the release-to-release compatibility contract.

This RFC proposes extending the existing test infrastructure to validate two downstream contracts:

Runtime contract : the released lemond behaves as expected by consumers.
Distribution contract : consumers can reliably obtain and launch the correct version of the embeddable lemond artifact.

The goal is to detect unintended breaking changes before a Lemonade release reaches downstream consumers.

A downstream integration can break even when Lemonade's existing tests pass.

For example:

Lemonade Changes

API behavior changes
→ Unsloth integration breaks

Artifact naming changes
→ Download fails

Release URL changes
→ Unsloth gets 404

Artifact contents change
→ Embedded lemond fails to start

Version/artifact mismatch
→ Downstream receives unexpected lemond

These are all part of the effective Lemonade integration contract.

Primary goals

• Reuse and strengthen existing Lemonade API/integration tests.
• Identify which existing tests represent the supported public contract.
• Detect unintended API breaking changes between releases.
• Validate embeddable artifact URLs and contents.
• Validate representative downstream integrations, initially Unsloth + embedded lemond.
• Make compatibility testing a release gate.

Non-goals

• Replacing existing unit/integration tests.
• Testing every possible downstream application.
• Guaranteeing compatibility with undocumented/internal APIs.
• Preventing intentional breaking changes.

High-level design

Compatibility Contracts and Testing

The compatibility contract includes both runtime behavior and distribution:

          Lemonade Release
                 │
      ┌──────────┴──────────┐
      │                     │
 Runtime Contract      Distribution Contract
      │                     │
  API behavior        URLs / artifacts
  schemas             packaging
  errors              version
  streaming           startup
      │                     │
      └──────────┬──────────┘
                 ▼
         Downstream App
            (Eg., Unsloth)

Tests should verify that:

• Supported API endpoints and request/response behavior remain compatible.
• HTTP errors, streaming, model lifecycle, and relevant semantics remain compatible.
• Version-specific embeddable download URLs continue to resolve.
• The downloaded artifact contains the expected lemond runtime.
• The artifact version matches the requested Lemonade release.
• A representative downstream integration continues to work

Where possible, the existing tests should be reused rather than duplicated.

Downstream Compatibility for a few Apps

Add a small number of consumer-level tests representing real integrations.
These tests should exercise the actual integration path used by Unsloth rather than a separate mock implementation. Additional downstream consumers can be added over time.

Breaking changes

None

Maintenance plan

• Lemonade team owns the compatibility suite and release gate.
• Update contract tests whenever a public API, artifact format, URL, or supported integration intentionally changes.
• Add a regression test for every downstream compatibility issue discovered after release.
• Review the compatibility suite periodically to remove obsolete coverage and add newly supported downstream integrations.
• Keep downstream tests lightweight and focused on critical integration paths to minimize CI maintenance and runtime.

Risks

None

[jeremyfowers]

Thank you for starting this discussion @iswaryaalex! Very important.

@danielholanda : would you kindly see if you can map the breaking-changes protections needed for Skills into this? IMO this is a much more maintainable way to provide protections for Skills than running Skills CI in the main lemonade repo.

[danielholanda]

Awesome! Will review and provide feedback.

[danielholanda]

Just took a look at the proposal and agree that this is an outstanding way of keeping downstream compatibility. I also agree that Lemonade skills can be seen as a downstream app and their maintenance will be significantly simplified with something like this in place.

[jeremyfowers]

The artifact version matches the requested Lemonade release.

Where is this concern coming from? ie, would would it even take for us to break this? It is quite hardened.

[jeremyfowers]

On a related note:

The packaged lemond starts successfully.

This is not a new requirement, all of our existing CI tests already assert it. Can this be removed from this RFC or is there a new specific consideration?

[iswaryaalex]

Good to know, calling it out that we do some thorough testing already. Updated

[jeremyfowers]

@iswaryaalex I'm highly aligned to the concept of a contract. What would help us most at this juncture is to understand what that contract would look like. You have some specifics already, like "Version-specific embeddable download URLs continue to resolve."

Can we get a robust set of specifics here? Especially which APIs need to be protected.

I'd also like to see a plan for protecting the contract from accidental degradation. Imagine if some rando asked their agent to change how /v1/load works, and their agent modified both the C++ and the contract test in the same PR, then the contract test would come up green, even though the contract is being violated. One solution would be to place the contracts test definition in a folder with a narrow CODEOWNERS file.

[iswaryaalex]

1. Which API's to be protected:
   I could define 2 Tiers of APIs

• Tier 1: APIs and behaviors whose regression can prevent a supported downstream application from discovering, downloading, loading, managing, or using a Lemonade model.
• Tier 2: APIs provide supporting functionality used by downstream integrations but are not required for the fundamental application lifecycle.

2. Test plan :

• Dedicated tests/contracts/ with CODEOWNERS
• Release tests for compatibility in CI
• Downstream golden apps testing (need comments on this)

[jeremyfowers]

Tier 1: APIs and behaviors whose regression can prevent a supported downstream application from discovering, downloading, loading, managing, or using a Lemonade model.

These sound like the right APIs to protect via contract. Can you make the specific list?

Test plan SGTM

[danielholanda]

I'm also highly aligned with the idea of a contract and that those should be programmatically tested.

• Will model names be part of the contract?
• Beyond the embeddable lemond artifact, will Lemonade's other distribution channels (and how often each one is updated) be part of the contract? Example: winget artifacts being released every time Lemonade is released.
• Will there be a single, continuously updated file that downstream apps can use to check when contracts were broken?
• Is there a mechanism to alert downstream apps of contract breaks?

[jeremyfowers]

Will model names be part of the contract?

They really shouldn't be. Models go stale and should not have an infinite shelf life.

Beyond the embeddable lemond artifact, will Lemonade's other distribution channels (and how often each one is updated) be part of the contract? Example: winget artifacts being released every time Lemonade is released.

The winget is unofficial so it would never be part of the contract. Same goes for Arch and Homebrew.

The official channels: PPA, .deb, .rpm, .msi, and Snap can be part of the contract.

Will there be a single, continuously updated file that downstream apps can use to check when contracts were broken?

I would prefer a single source of truth: download and scan the breaking changes sections of each release at https://github.com/lemonade-sdk/lemonade/releases

Is there a mechanism to alert downstream apps of contract breaks?

Star the repo and you get notified of releases!

[danielholanda]

You know what would be really cool? This could be part of a "v2" of this:

Every line item in the contract could be explicitly numbered/registered. Apps could then (with the help of a script we author) list which parts of the contract they rely on and have a CI/CD test that programmatically checks those parts for upcoming breaking changes. The great thing is that downstream users wouldn't have to care about breaks in parts of the contract they don't rely on.

[iswaryaalex]

What if we identified a few Golden apps downstream and particilarly track the APIs they are using and ensure they don’t break?

Most of them (unsloth as a casestudy) rely on discovering, downloading, loading, managing, or using a Lemonade model.
Some of them use what I am calling as Tier 2s here #3732 (reply in thread)

[jeremyfowers]

What if we identified a few Golden apps downstream and particilarly track the APIs they are using and ensure they don’t break?

Yes please do!