Preload trampoline binaries to avoid race conditions when lazily downloading during parallel builds

[dnagoda]

WHY are these changes introduced?

shopify app build fails intermittently on CI with ETXTBSY ("text file busy") when building apps with multiple Shopify Function extensions:

spawn ETXTBSY
Command failed with ETXTBSY: .../node_modules/@shopify/cli/bin/shopify-function-trampoline-1.0.2 -i <wasm> -o <wasm>

The trampoline binaries (shopify-function-trampoline-1.0.2, shopify-function-trampoline-2.0.1) are not shipped with the npm package — they are lazily downloaded from GitHub on first use by downloadBinary(). When renderConcurrent runs parallel extension builds, multiple concurrent tasks race to download the same binary. One task opens the file for writing (moveFile with {overwrite: true}) while another tries to execute it, and the Linux kernel returns ETXTBSY.

The in-process dedup map (downloadsInProgress) prevents redundant downloads within a single event loop tick, but the race window between the download completing and the file being executed by a concurrent build is still open.

This is the same class of bug that was previously fixed for Javy binaries in #2877 — which introduced installJavy() to pre-download Javy + plugin binaries before parallel builds start. The trampoline was simply missed because it was added later.

WHAT is this pull request doing?

Adds installTrampolines(app) — a new pre-download function that mirrors the existing installJavy(app) pattern. It eagerly downloads both trampoline versions (v1 and v2) before the parallel build fan-out, so that when individual runTrampoline() calls run concurrently, the binaries are already on disk and downloadBinary() is a no-op.

Key design choice: Unlike Javy (where the version depends on each extension's @shopify/shopify_function package version), the trampoline version is determined post-build from WASM imports (shopify_function_v1 vs shopify_function_v2). Since there are only two possible versions and the binaries are small, we simply pre-download both for any app with function extensions.

Changes:

• packages/app/src/cli/services/function/build.ts — Added installTrampolines(app): skips if no function extensions, otherwise downloads both V1 and V2 trampoline binaries in parallel
• packages/app/src/cli/services/build.ts — Calls installTrampolines alongside installJavy (via Promise.all) before renderConcurrent
• packages/app/src/cli/services/deploy/bundle.ts — Same pre-download before parallel deploy builds
• packages/app/src/cli/services/dev/processes/draftable-extension.ts — Same pre-download before dev draft updates
• packages/app/src/cli/services/function/build.test.ts — Added 2 tests: verifies both versions are downloaded when function extensions exist, verifies no downloads when they don't

How to test your changes?

1. Unit tests: pnpm vitest run packages/app/src/cli/services/function/build.test.ts (28 tests pass including 2 new)
2. Manual repro (requires Linux CI or Docker):
   • Create an app with 2+ function extensions
   • Clear any cached trampoline binaries from node_modules/@shopify/cli/bin/
   • Run shopify app build — should succeed consistently without ETXTBSY
     • This is hard to reproduce manually and is flakey on CI, but an example of the failure can be found here: https://buildkite.com/shopify/checkout-blocks-tests/builds/5475#019de03e-9d05-470f-9864-765465ea71a7
3. Regression check: existing Javy pre-download behavior unchanged — installJavy and installTrampolines run in parallel via Promise.all

Checklist

• I've considered possible cross-platform impacts (Mac, Linux, Windows)
• I've considered possible documentation changes
• I've considered analytics changes to measure impact
• The change is user-facing — I've identified the correct bump type (patch for bug fixes · minor for new features · major for breaking changes) and added a changeset with pnpm changeset add

[dnagoda]

• Preload trampoline binaries to avoid race conditions when lazily downloading during parallel builds #7438 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[dmerand]

Have you tested dev with this change? I wonder if the dev-side preload is attached to the wrong lifecycle point, meaning this likely still misses app dev startup in some cases.

Right now the new preload runs in pushUpdatesForDraftableExtensions() in packages/app/src/cli/services/dev/processes/draftable-extension.ts, but the actual initial concurrent extension builds happen in AppEventWatcher.start() via buildExtensions(...).

Two consequences:

• setupDevProcesses() uses setupDevSessionProcess(...) instead of setupDraftableExtensionsProcess(...) when developerPlatformClient.supportsDevSessions is true, so that path appears to get no trampoline preload at all.
• More generally, the watcher startup is the place that owns the initial build fan-out, so putting the preload in a side process feels racy / easy to bypass.

Would it be safer to move this into the shared watcher startup path (e.g. before AppEventWatcher.start() begins buildExtensions(...)), so all app dev modes get the same protection?

[dnagoda]

I'm very low context on CLI architecture, but I can definitely look into how we could do this at a higher level in the flow. Would we want to move the javy preload at the same time?

[dmerand]

I'm also low-context on the functions side of things. Perhaps it works for Javy at this layer, it should work for Trampoline. Hopefully somebody with more Functions experience can weigh in.

[dnagoda]

I think it makes sense to move both preloads out of the watcher. There's probably also an argument that we don't need the preload for dev contexts as these are likely to run in long lived environments. The issue with the build and bundle flows is that they would often be run in CI on fresh env where the resource has to be downloaded fresh.

[dnagoda]

Moved to SetupDevProcesses.

[dmerand]

One compatibility concern here that my 'bot found: installTrampolines() now eagerly downloads both trampoline versions, but trampolineBinary(version) only marks Windows-on-ARM support for versions >= 2.0.1.

That means on win32/arm64, a function app that only needs the v2 trampoline may have worked before (because runTrampoline() lazily selected/downloaded just v2), but could now fail up front trying to predownload the unused v1 binary.

Would it be safer to skip pre-downloading versions that are unsupported on the current platform/arch, or otherwise avoid failing the whole command on an unused trampoline version?

[dnagoda]

Yes, that makes sense. Let me see how we check for the needed version and look into adding that logic here.

[dnagoda]

We can't determine which trampolines we need during the preload because we don't have WASMs to introspect, but we can protect from download errors and log. This will protect us from having issues during the preload. Thanks for catching this. 👍

[jeffcharles]

I've opened #7443 as an alternative fix. The underlying problem IMO is downloadBinary returning too early. That is, when it sees the file on the file system, not when the file is ready to be executed. What I prefer with my fix is it'll cover all Functions binaries, we don't have to download more than we need to, and we don't have to swallow errors during the download process.

If we do opt to go with this PR, I would ask we don't swallow all errors that happen with downloading but much more selective. The error messages would be misleading if someone tries building a Function for the first time and GitHub is having an outage or if they're using the v1 trampoline on Windows for ARM.

[jeffcharles]

Thinking about this a little more, if the Shopify CLI, or the Functions binaries, were to drop support for Windows 10, we might be able to use Prism on Windows 11 to run the Intel binaries on ARM machines. I haven't tested it though as I don't have access to a Windows 11 machine.

[dnagoda]

@jeffcharles Thanks for taking a look. Yes, your approach is much better. I completely missed the downloadsInProgress behavior. Closing this PR is favor of your approach.