RFC: New release process

[jeremyfowers]

Updates

1. Added more release lifecycle information per @abn's questions.
2. Big changes to the release lifecycle and administration per many comments.
   • stable releases are tagged by a human, and can be skipped by not tagging
   • 100% deterministic versioning that is consistent across sense across stable, candidate, bleeding-edge, PR, and from-source builds.
3. More feedback: 4-digit years, refined the post ~ versioning, more specifics about timing.
4. Shorted the candidate label and moved the revert/hotfix workflows to future work.

Proposal Size

Major feature (spans multiple well-scoped PRs)

User Story

Lemonade's users and maintainers want our stable releases to be more stable. After a lot of deliberation, we decided that the path to this is more testing (as opposed to stricter PR review).

1. New Lemonade releases will undergo community testing in a release-vX.Y.Z branch for 1 week before being released to the stable/latest channels.
2. Release candidate installers will be provided in new candidate (PPA, Snap) and prerelease (GitHub) channels for Ubuntu, Windows, Docker, macOS, Fedora, and Debian.
3. The release lifecycle will be automated to the greatest extent possible (see High-Level Design).
4. New branch management policy: commits that get a lot of negative feedback during release candidate testing will be reverted out of main and release-v* and sent back to PR for additional changes and review.

This plan will only work if a substantial amount of people get involved in testing, so please comment/vote if you are willing to help out!

High-Level Design

Release lifecycle

1. Releases are branched from main and tested over the course of 1 week.
2. Issues identified via testing should be resolved by:
   a. "Wont fix" in this release: issue is deemed minor / non-breaking enough to be fixed on main and solved in a future release.
   b. Hotfix: blocking, yet easily fixable, issues should be resolved by merging a fix into main and then cherry-picking it into the release branch.
   c. Revert: blocking issues that would be excessively complex to fix should have their commit(s) reverted out of both main and the release branch.
3. At the end of 1 week, the release branch out to be stable enough for a stable release.

New versioning system

The versioning system for Lemonade should move from marketing-based to date-based to facilitate the automated systems (see below). The justification is that everything will be much easier to automate if the version number is deterministic.

Current system: major.minor.patch where major/minor change depending on our hype level (it's pretty arbitrary!).

New system: year.week.number.

• year is the 4-digit year of the release, which is 2026 right now. If Lemonade lasts 75 more years we'll come up with a new system :)
• week is which week of the year of the release, indexed 1 to 52. This RFC was written on 8 September, which is in week 37 of 2026. Releases are branched at 7 pm UTC on Wednesdays.
• number is the number of commits on the release branch, after it was branched from main.

For example: if this policy was already in effect and we branched at 7 pm UTC on 9 September 2026:

1. the branch name would be release-v2026.38, the year.week for next week's release.
2. the first release candidate would be v2026.38.0. If that artifact passes testing, we'd release the v2026.38.0 artifact to stable. If not, we need another candidate artifact.
3. If we fix a bug and need a second release candidate, that would be v2026.38.1.
4. If we release v2026.38.1, but later realize we need to hotfix it in stable, the hotfix release would be v2026.38.2.

Note: the MSI version number will be trimmed to 2 digits, 2026 -> 26

The number field is always incrementing monotonically, and always refers to exactly 1 artifact regardless of whether we release that artifact to stable.

Important: we will also remove the version number from CMakeLists.txt entirely. year.week.number is 100% deterministic, so the build system will apply the version number at build-time based on the branch name (which contains year.week, see below) and the number of commits.

Versioning non-candidate builds

Non-candidate builds, for example build-from-source, PR checks, or artifacts for the bleeding-edge channel, will be versioned year.week.0~$(number of commits on branch).$(hash), where:

• week is the week where this commit would release
  • if a release branch hasn't been made yet this week (e.g., a build before 7 pm UTC on Wednesday), week is next week.
  • after 7 pm UTC on Wednesday, week is 2 weeks from today, since the commit missed the cutoff time for next week's release.
• branch is main for bleeding-edge, or whatever branch you happen to be building from.

For example, a build of a commit made to main on 10 September 2026 would have week=39 and a version like v2026.39.0~1595.ff22950d.

You can also supply a .version file to overwrite the built version to anything you like.

Release admin

The role of the "release admin" is changing since most of their work is being automated away and new responsibilities are being introduced.

The release admin will:

1. Moderate a new #release-candidate discord channel. Inform the testers when new RCs are available and instruct them on what to test (using repo-manager auto-generated to-do lists as a guide).
2. Monitor any issues with the candidate label.
3. Decide if any actions need to be taken: cherry-picking to release branch or reverting a commit.
4. Write the release announcement that will be posted to Discord (using the repo-manager auto-generated announcement as a guide.)
5. Tag a commit on the release branch to ship it to stable. The tag must match the version number of the build for that commit.

In the (hopefully rare) event that a release branch still has blocking issues on Friday at 7 pm UTC, that week's release should be skipped and we will try again the next week. If this happens we should hold a postmortem and decide if the PR review and/or release policies need to be revised (e.g., do we need 2 weeks of testing?).

Finally, we should also work on expanding the release admins group beyond the current 4 individuals (@jeremyfowers @kenvandine @ramkrishna2910 @superm1 ) .

Automated Systems

Automated Release

These automations are intended to free up admin/maintainer time, so that this time can be spent on testing and taking corrective actions instead.

On Wednesday, at 7 pm UTC (2 pm ET) every week, the following actions will take place automatically via a GitHub Actions cron job:

1. main is branched into release-v$(year.)$(week+1), where week+1 is next week's index (since the release is going out next week)
2. The candidate channels of the PPA and snap, along with the prerelease channel of the repo, will release v$(year.)$(week).0 artifacts for testing, year and week are derived from the branch name.
3. Any time a release-v* branch gets a new commit, a new v$(year.)$(week).$(number) artifact ships automatically to the candidate/prerelease channels.

Future work: Branch Management Tools

In the future, but not in the initial set of PRs, we may introduce the the following scripted actions to help maintainers with common tasks:

• Cherry-pick to release branch: cherry-picks one or more commits from main in the current release branch release-v*. Intended to deliver fixes only, not features.
• Release hotfix: cherry-pick one or more commits into any release-v* (which triggers new candidate artifacts and option to tag them into stable).
• Revert commit: reverts one or more commits from both main and the current release-v* branch.

Breaking Changes

See the "new versioning system" section.

Maintenance plan

This new system is expected to be self-maintaining after it is merged, other than the work of the release admin noted above.

Risks

1. Automatically creating the release branches has the potential to include or exclude commits that people would have had other intentions for; hopefully people learn how to work around the schedule.
2. Reverting commits can lead to bad emotions, however IMO this is better than frantically trying to hotfix something that has fundamental or complex issues.
3. One week may be too work for a volunteer testing group to discover and hotfix/revert all problems; if this is the case can extend the cycle, organize better, or recruit more testers.

@superm1 @ramkrishna2910 @kenvandine @sawansri @Geramy @bitgamma

[abn]

Adding my 2c.

The RFC doesn't explicitly say what happens when the RC week actually finds something. The tools imply an answer. Cherry-pick pulls from main and revert hits both branches, so presumably it is a fix forward in main, cherry-pick into release-v*, never commit to the release branch. Either way it would be good to be explicit about the intent.

The Wednesday cron ships release-vY.W.0 to stable whether or not a blocker is open, and none of the release admin's four duties is deciding whether to ship. Can a release be held? Can one be skipped outright, no v26.38.0 at all, if the RC is broken? What happens to the numbering if it's held too long?

Cherry-picks also conflict, more often than one might expect. Resolving one means a person or an agent writing code that never existed in main and never went through PR review. Does the workflow fail back to a human? Does the resolution get reviewed, and does it get a follow-up PR against main so the next weekly doesn't regress? Same for hotfixes: cherry-picked from where, and re-landed how?

I also wonder where feedback goes when it uncovers a use case we don't currently test. Is there a plan for an e2e suite that could stop the same class of issue recurring?

The "a lot of negative feedback" criterion is the other thing I'd want nailed down. I imagine the release admin triages the issues and decides what's actually a blocker, rather than it coming down to whoever is loudest?

[jeremyfowers]

The RFC doesn't explicitly say what happens when the RC week actually finds something.

True! It's already documented in release.md and isn't changing here, but I added these details since almost nobody has had to read release.md.

The Wednesday cron ships release-vY.W.0 to stable whether or not a blocker is open, and none of the release admin's four duties is deciding whether to ship. Can a release be held?

Added this policy to the discussion body. I am basing my policy proposal on the idea that this should never happen; between our RFC, PR review, and week-long testing it should not. If people think 1 week of testing doesn't give enough confidence let's extend it to 2 weeks for a start, and then reduce to 1 week if nothing ever happens in the second week.

Cherry-picks also conflict

Technically true, but only minimal hotfixes should be cherry-picked.

I also wonder where feedback goes when it uncovers a use case we don't currently test.

People should file issues/RFCs to fix/add to the existing CI suite.

I imagine the release admin triages the issues and decides what's actually a blocker, rather than it coming down to whoever is loudest?

The release admin should make the call, but ultimately this should not be a hard decision. Lemonade's quality and ease of use are tier 1 concerns laid out by the project philosophy, and trump any new feature. If even one person is loudly complaining that their experience or workflow has been broken that's a powerful signal.

[abn]

Technically true, but only minimal hotfixes should be cherry-picked.

Well my point was more that even minimal fixes may be layered on top of current state of main which might include bigger changes.

Edit: in Poetry 90% (ballpark) were handled by a label triggered backport bot which opened it's own pr for review. The label was applied to the PR we wanted to backport into the release branch. But this was largely because we maintained y-stream.

If even one person is loudly complaining that their experience or workflow has been broken that's a powerful signal.

Personally, I feel that is a slippery slope. The question really should be assessed against a set of what use cases lemoande support and at what levels. As an example, the removal of env vars completely breaks my workflows; I can definitely start making a fuss about that but in my case I just chose to workaround the issue instead of making noise. 😇

That said, this is definitely not a blocker by any means.

[jeremyfowers]

The Wednesday cron ships release-vY.W.0 to stable whether or not a blocker is open

Thanks for the feedback on this, shipping to stable is back to being a human gate.

[superm1]

I like to automate where possible but I think the place I worry about is if there is a user facing change that affects their configuration data. For a hypothetical example if we had a schema bump in config.json happen in rc1 and it caused problems and we reverted it, an rc2 is created. What happens to a tester who was on rc1 and now the new schema?

I suppose we need to be extra careful with those kinds of changes.

[jeremyfowers]

The only real answer here is that people need to back up their config and models before they install RCs.

However, as of #3521 we're also requiring an RFC for breaking changes, and this should help us be extra careful with review and rollout of such changes.

[PCAssistSoftware]

Perhaps Lemonade should include a function to allow the config files at least to easily be backed up? Personally I have a simple script I run regularly to back them up to folder on desktop, just in case. But I can certainly see benefit of a built in backup for less technical users etc

[superm1]

Well we could trigger an automatic backup on schema migration. Maybe that's the right answer.

[jeremyfowers]

Well we could trigger an automatic backup on schema migration.

A+ comment to leave on the first RFC to propose a schema migration!

[jamiej]

If testing can be automated and automation were short (30-60min / day), then I'm open to it.

My target env is headless Proxmox w/ 3xR9700. Can likely re-target my proxmox-community script installer (https://github.com/community-scripts/ProxmoxVED/blob/main/install/lemonade-server-install.sh and https://github.com/community-scripts/ProxmoxVED/blob/main/ct/lemonade-server.sh) to the release-candidate channels. That makes it easy to spin up an new or update an existing LXC for Lemonade w/ GPUs.

Upgrade scenarios,

• Release -> RC1
• Release -> RC2
• RC1 -> RC2 ...

Can be accomplished a couple different ways using the same proxmox community-scripts paths. I'm willing to make those real on my end if the value prop is good for Lemonade devs, e.g. testing limitations of both automation and target environment.

[jeremyfowers]

If testing can be automated and automation were short (30-60min / day), then I'm open to it.

As a baseline it would be amazing to have 1 tester per platform just install the RCs and do anything with them. A lot of recent breaks would have been caught that way, as our main gap was that nobody was trying the upgrade path for all the installers.

It's also helpful if people can try their production workloads on the RCs, but not everyone needs to do that.

[PCAssistSoftware]

I personally plan to install and use the RC on my main system so it will get a full test each time against my normal production / daily workloads. So you have one Ubuntu / Strix Halo user ready here :)

[jamiej]

It's also helpful if people can try their production workloads on the RCs, but not everyone needs to do that.

That's possible. Sketch would be a script to clone LXC, apply RC update via community script or fetch and apply directly from RC repo, launch smoke (pre-loads models) then patch DNS. From there Honcho, Hindsight, and Hermes instances would hit the endpoint, telemetry + metrics (unless RC broke it) captured by Phoenix. Finally run memory replay to exercise the RC and all GPUs.

Fairly confident my primary agent can orchestrate the entire workflow. Grok has been solid for this type of IT drudgery, and more. If this RFC becomes real, I'll have the agent draw up the above plan and tackle it step-by-step.

Caveats for my setup

• Coverage is llamacpp + rocm backend, none of the other Lemonade engines or back-ends.
• No dynamic swapping of models. Models are pre-loaded and pinned to their assigned GPUs, e.g. ROCm0...2

I do have some out-of-band test targeting Lemonade vLLM, but nothing production. Experimental vLLM version is too old, and latest +dev for gfx120X explodes. I've a separate LXC for vLLM hosting, exercised AITER paths today.

[superm1]

BTW - Happy on the proposed version scheme.

[jeremyfowers]

I though you might be :)

[jeremyfowers]

Thinking through a rollout plan for this...

1. I want @superm1 @kenvandine @ramkrishna2910 to reply APPROVE here since they're the other release admins at the moment.
2. Then I would make a PR to update release.md and add the basic automations around tagging and releasing from GitHub (perhaps left dormant until the next steps are done)
3. @superm1 : enable the candidate PPA
4. @kenvandine : enable the candidate channel on the Snap
5. Follow ups by me: implement the additional automations for things like cherry picking and hotfix releases.

I will also need to investigate what can be fully automated vs. what would need to be fully or quasi-manual, considering main and release-v* are protected branches.

[superm1]

@superm1 : enable the candidate PPA

#3529

I want @superm1 @kenvandine @ramkrishna2910 to reply APPROVE here since they're the other release admins at the moment.

I don't understand what how we get Launchpad builds to stable PPA at this moment. Is it a copy of the binaries from candidate? Or is it a totally separate upload? Or is it a manual process?

[jeremyfowers]

I don't understand what how we get Launchpad builds to stable PPA at this moment. Is it a copy of the binaries from candidate? Or is it a totally separate upload? Or is it a manual process?

My thinking... and feel free to disagree...

1. candidate is automatically updated whenever a new commit hits the release branch. Same as how bleeding-edge is automatically updated whenever a new commit hits main branch.
2. Nothing changes for stable: when we tag a commit on the release branch, that commit is released to stable.

[superm1]

So that only works if no tags for release branch until release. If you agree with that, it's fine.

[superm1]

Adjusted the PR in #3529

[jeremyfowers]

Just pushed a big update to the RFC body to address this

[ramkrishna2910]

RFC makes sense to me.
I guess we will find more as we go through this process a couple of times. I really hope we dont have to go down to a 2 week cadence.
The one wild card here is the volume of testing we get from volunteers.

[jeremyfowers]

Look at #release-candidate on the discord! Lots of people signed up already <3

[superm1]

year is the 2-digit year of the release, which is 26 right now. If Lemonade lasts 75 more years we'll come up with a new system :)

Why not just make it 4 digits from the beginning? It will be a lot more obvious it's a year then to people.

will be versioned year.week.0~$(number of commits on branch)$(hash),

Since some hashes can start with a number I would suggest this has a delimitter for num commits and the hash like this:
year.week.0~$(number of commits on branch).$(hash)

week is the week where this commit would release.

It's implied but I think you should make it explicit that this is based off the branch date in the middle of the week.

[jeremyfowers]

year is the 2-digit year of the release, which is 26 right now. If Lemonade lasts 75 more years we'll come up with a new system :)

Why not just make it 4 digits from the beginning? It will be a lot more obvious it's a year then to people.

MSIs dont allow 4 digit numbers but we can trim it. I'm not opposed. I agree 2026 is vastly more meaningful than 26 to an untrained eye.

[jeremyfowers]

Why not just make it 4 digits from the beginning?

Done, pushed.

I would suggest this has a delimitter for num commits

It was just a typo in one spot, fixed, pushed.

It's implied but I think you should make it explicit that this is based off the branch date in the middle of the week.

Clarifications pushed.

[superm1]

reply APPROVE

Instead of APPROVE, how about we approve by emoji? You can count lemons.

🍋

[ramkrishna2910]

🍋

[kenvandine]

🍋

This looks great to me. I'm sure we'll tease out a few things as we transition to this, but that's to be expected. We are already automatically publishing release-candidate builds of the snap to the candidate channel. Although that's currently triggered by the repo-manager workflow. Do we want to move those to a common workflow?

[jeremyfowers]

Approved 3/3!