feat: add support for git dependencies by gonzamontiel · Pull Request #311 · digital-asset/dpm

gonzamontiel · 2026-08-21T16:27:33Z

Why

dpm already resolves remote DARs from OCI. This change adds Git as a second source for pre-built .dar files, using the existing add / install / update / resolve lifecycle.

Git DARs may be declared in dependencies or data-dependencies. Mutable refs are pinned to a commit SHA in the field where they were declared. resolve then returns the cached local path (resolved-dependencies or resolved-data-dependencies).

Summary

In daml.yaml, a repo-file Git DAR is declared as git:<host>/<owner>/<repo>#<ref>?path=<file>.dar.

Example:

artifact-locations:
  "@example-repo":
    url: "git:github.com/org/repo"

dependencies:
  - git:github.com/org/repo#main?path=packages/foo.dar
  - "@example-repo#main?path=packages/bar.dar"

data-dependencies:
  - git:github.com/org/repo?release=v1.0.0&asset=foo.dar

On install / update, branch and tag refs are rewritten to a 40-character commit SHA in the same YAML field. @alias lines are authoring shorthand: pinning expands them to a full git: URI. Fetched DARs land under ~/.dpm/cache/git/<host>/<org>/<repo>/<commit-sha>/<repo-relative.dar> (release assets use a hash of tag+asset as the ref segment). resolve returns that absolute path; it does not fetch.

Repo-file fetch is any HTTPS Git host. There's also a release form ?release= which is a GitHub-only feature, since it relies on api.github.com.

Technical design

This collapsed section is the implementation and review topics for this change. It covers the feature's core (parse, fetch, pin, and resolve, including data-dependencies), lock and update, GitHub releases, and UX extras.

Details

Git dependencies core

This section implements the foundational capability: reference a pre-built .dar at a path inside a Git repository, covering parse, fetch, pin, cache, and resolve. Git DARs in data-dependencies use the same path; they land in a different YAML field and in imports.resolved-data-dependencies.

Scope

Parsing & validation (pkg/gitparse: parse.go, validate.go): the canonical one-liner git:<host>/<owner>/<repo>#<ref>?path=<file>.dar. Parsing also rejects mixed release / ref / path / asset, absolute / .. paths, non-.dar paths, and non-HTTPS clone URLs (SSH is rejected). file:// is test-only (DPM_TEST_ALLOW_FILE_GIT).
Fetch & cache (pkg/gitpuller + go-git): clone or reuse …/<repo>/.work, check out the ref, copy the .dar atomically. Empty source files and empty cache files are treated as misses. Symlinks that escape the worktree are rejected (RejectSymlinkOutsideRoot).
Cache layout (pkg/assistantconfig): cache/git/<host>/<org>/<repo>/<commit-sha>/…. Segments are sanitized against traversal.
CLI lifecycle: dpm add dar <git-uri> <--dependencies | --data-dependencies>. Exactly one flag is required. install package and resolve walk both fields. Pinning rewrites only the field that contained the entry.
Pinning: mutable refs are resolved and rewritten to a commit SHA in the declared YAML field at install time.
Aliases: artifact-locations may hold a bare git: repo URL (no #ref or query). #<ref>?path= / ?release= stay on the dependency line. A location URL that already carries # or ? is rejected.
Resolve: unpinned (mutable) refs and missing cache entries fail; they are not fetched. Git data-deps appear only under imports.resolved-data-dependencies.

Lock + update

Reproducibility and update semantics layered on top of the core.

Scope

Lockfile integration (GitLockKey / GitLockKeyForDep in pkg/gitparse): stable Git identity keys when DPM_LOCKFILE_ENABLED=true, same gate as OCI. computeExpectedLockfile still only records dependencies, not data-dependencies.
dpm update: re-resolves mutable refs and rewrites the pin in that field. Already-pinned SHAs are not re-resolved; they are fetched only if the cache is missing.
Release assets are expanded/fetched in PrepareGitDependencies (both fields) before the per-DAR update loop. The Git DAR updater does not rewrite release tags.
dpm update --check is implemented (see UX extras); it is not part of the mutating update path.

GitHub releases

A second transport for teams that distribute DARs as GitHub Release assets rather than in-tree files. Works in both dependencies and data-dependencies.

Scope

Releases API client (pkg/githubrelease): list and download release assets. Host is github.com only (plus a test override). GitLab / ?release= is rejected with a pointer to #<ref>?path=. daml.yaml is left unchanged on that rejection.
Expansion (pkg/damlpackage/gitrelease.go): a release line without an explicit asset expands into one dependency per .dar asset in the release. Expansion preserves wrapper shapes (main-package-id) and does not duplicate assets already listed.
Expansion failure rolls back daml.yaml. If expansion succeeds and a later download fails, the per-asset lines stay so retry does not re-expand.
Bulk fetch (pkg/gitpuller/release_fetch.go): download missing release assets during add, install, and update.
Release cache path: cache/git/<host>/<org>/<repo>/<sha256(tag+"\0"+asset)>/<asset>.

dependencies:
  - git:github.com/org/repo?release=v1.0.0             # expands to all .dar assets
  - git:github.com/org/repo?release=v1.0.0&asset=foo.dar

data-dependencies:
  - git:github.com/org/repo?release=v1.0.0&asset=foo.dar

UX extras

Ergonomics and operability that are convenient but not required for a functional feature.

Scope

Input normalization (gitnormalize): coerce alternative inputs — git:https://…, host-first shorthand, GitHub …/blob|raw/<ref>/…, and GitLab …/-/blob/<ref>/… — into the canonical one-liner. Well-known hosts for scheme-relative recognition: github.com, gitlab.com, bitbucket.org, codeberg.org. Other HTTPS hosts work when written as git:<host>/…#ref?path=….
dpm update --check: both fields. Passes when repo-file deps are SHA-pinned and cached, and when release assets are cached. Fails on mutable refs, missing cache, or an unexpanded ?release= line (no asset=). Non-mutating.

Coverage

The new modules pkg/gitparse, pkg/gitpuller, and pkg/githubrelease each include its own set of unit tests. Testing involving the CLI was included in dars_test.go, dpm_add_test.go, install_package_test.go, and update_git_test.go. Roughly 2.9k lines of this work is meant for coverage.

You can run the following commands to run the related tests:

go test ./pkg/gitparse ./pkg/gitpuller ./pkg/githubrelease \
        ./pkg/damlpackage ./pkg/assistantconfig \
        ./pkg/resolver ./cmd/dpm/cmd/update

go test ./cmd/dpm/cmd -run 'TestSuite/Test.*Git'

Testing

For ease of functional testing, we have prepared a demo app dpm-git-links-demo. You can checkout, run the README sequence, and play around with it.

dpm install package
dpm build
dpm test

dpm update re-resolves and re-pins refs. samples/ has the other YAML shapes: data-dependencies vs dependencies, artifact-locations aliases, and GitHub release assets.

hrischuk-da · 2026-08-21T17:16:19Z

 ::

-  dpm add dar <oci-uri> <--dependencies | --data-dependencies> [flags]
+  dpm add dar <oci-uri|git-uri> <--dependencies | --data-dependencies> [flags]


Please update the dpm docs at https://github.com/canton-network/cf-docs.

Hey @hrischuk-da , thanks for the comment! the PR for the updated docs is here canton-network/cf-docs#1463

sammy-da · 2026-08-27T20:32:22Z

+	GitRef     string
+	DarPath    string
+	CloneURL   *url.URL
+	GitRelease bool


these four fields all go hand-in-hand, right?
if so i would group them under one struct

hey @sammy-da, yes, these are meant to play together, and you're right, it's cleaner having them into one struct. We'll make that change : )

Addressed in 0ef4cd6

sammy-da · 2026-08-27T20:37:45Z

there seems to be a substantial amount of code to parse the git: urls.
i think it's big enough to warrant having its own package.

(What would be more ideal is to use an existing scheme or a library for this, if one already exists out there, rather than having to manually write this)

(for example https://github.com/hashicorp/go-getter seems to have git:: scheme)

fair point on the size! it slowly grew and it’s true that now it deserves it’s own gitparse package, we’ll extract it : )

on go-getter, we looked at it but chose not to use it for two reasons: the first one being that it uses a different URL syntax than the one proposed, and most importantly, it doesn’t support parsing for GitHub release links.
in our case, adopting such a tool would mean maintaining two types of Git-line formats, and we thought that would be cumbersome

Addressed in 773d773

sammy-da · 2026-08-27T20:57:39Z

+// GitStructuredFields is the structured git dependency form in daml.yaml.
+type GitStructuredFields struct {
+	URL     string `yaml:"url"`
+	Ref     string `yaml:"ref"`
+	Path    string `yaml:"path"`
+	Release string `yaml:"release"`
+	Asset   string `yaml:"asset"`
+}


i believe the damlc compiler right now does not allow dependencies that are of map type in daml.yaml.

The existing code here was trying to add support for that on the dpm side for the purpose of attaching main-package-id field onto every dar specified in daml.yaml (and yes that would ideally include dars sourced from git), but that's an unfinished feature at the moment.

I would just drop this "structured git fields" feature for now, and go with the string based approach

also fair, the feature would be really nice if damlc was aligned with it, but for the time being this syntactic sugar is gonna be gone as soon as we rewrite the line to be compliant with damlc, so yeah, we can drop it for now 👌

Addressed in 7ae6aa6

A 3.6 snapshot version of damlc has been released which now supports this syntax, and a formal 3.6 release should be coming shortly by end of month.

hey @dasormeter , thanks for pointing that out, do you think it's worth it to re-add the feature to this work? maybe it's better to include it as a follow up PR as this one is already big enough, wdyt ?

tomimor · 2026-09-21T16:03:34Z

Hi @hrischuk-da & @sammy-da! We would love to hear if you have any further feedback. Any visibility into the review timeline would be greatly appreciated as well.

We also compiled technical design decisions, feature descriptions, and testing at https://moonsong-labs.github.io/dpm/. This should add more context to this PR and hopefully help with the review process.

Thanks in advance!

…es-support

dasormeter · 2026-10-02T18:29:40Z

@ffarall okay now allowed for CI run of latest commit

There look to be a few tests failing on windows -- can you take a look?

you should be able to repro by running on runs-on: windows-latest on your own fork if you don't have a window machine handy to repro on
https://github.com/digital-asset/dpm/actions/runs/37046514981/job/110970327059?pr=311

There also is something wonky with perms on the Check workflow changes / workflow-policy (pull_request) check -- that is not blocking as of now but I'll try to fix separately. (that is getting fired because of the lockfile change)

ffarall · 2026-10-02T18:31:58Z

@ffarall okay now allowed for CI run of latest commit

There look to be a few tests failing on windows -- can you take a look?

you should be able to repro by running on runs-on: windows-latest on your own fork if you don't have a window machine handy to repro on https://github.com/digital-asset/dpm/actions/runs/37046514981/job/110970327059?pr=311

Thanks @dasormeter ! On it right now.

dasormeter · 2026-10-02T19:31:09Z

There also is something wonky with perms on the Check workflow changes / workflow-policy (pull_request) check -- that is not blocking as of now but I'll try to fix separately. (that is getting fired because of the lockfile change)

the workflow policy had a syntax error which has now been corrected, so just make sure to rebase before you do your next commit on this PR

dasormeter · 2026-10-02T20:26:49Z

Please make sure to update UNRELEASED.md with a user-friendly description of this new feature as part of this PR

…encies-support

dasormeter · 2026-10-05T13:08:02Z

@ffarall okay now allowed for CI run of latest commit
There look to be a few tests failing on windows -- can you take a look?
you should be able to repro by running on runs-on: windows-latest on your own fork if you don't have a window machine handy to repro on https://github.com/digital-asset/dpm/actions/runs/37046514981/job/110970327059?pr=311

Thanks @dasormeter ! On it right now.

Windows test still looks to be failing after kicking off run of latest commit
https://github.com/digital-asset/dpm/actions/runs/37261358880/job/111759809275?pr=311

Copilot

Copilot review overview

🟡 Changes recommended

Unresolved correctness issues affect dependency installation, URL round-tripping, and lockfile reproducibility.

Review effort: Balanced
Findings: 2 · 9

Open (11)

Preserve HTTPS ports and .git suffixes during Git URL formatting · New
Preserve ref-specific SHAs when regenerating Git locks · New
Do not publish empty DAR downloads · New
Invalidate lockfiles when mutable dependency selectors change
Make dependency expansion transactional across fields
Resolve HEAD consistently for reused repository clones
Reject ref and path selectors with release queries
Escape and parse @ characters in Git refs correctly
Preserve HTTPS scheme for ports and IPv6 authorities
Reject empty HTTP responses before publishing DARs
Resolve asset-bearing release aliases before deduplication

Resolved since last review (2)

Use unique temporary files for concurrent cache writes
Serialize concurrent repository worktree operations

dasormeter · 2026-10-05T18:41:28Z

The go.mod dependency changes have been separately incorporated on main, so you can now rebase this PR on main and will not get the warning on unexpected dependency changes in a fork contribution

#323

…encies-support

ffarall · 2026-10-05T19:58:52Z

@dasormeter Thanks for the review again! Much appreciated.

Windows tests failures: the previous test failures were fixed and checked in our fork's CI run. The fix for AtomicWriteFile in 0b73c56 introduced another test that launches 8 writes of the same out.dar file. Each of the 8 write their own temp file (as the fix implemented), and then all of them try to rename the temp file to out.dar, essentially racing for it. The Windows test CI passed in our run, but didn't here in your latest run, thankfully surfacing the race condition.
- On Unix, the rename is done in one step and each process writes in turn, being the remaining out.dar the one who renamed last.
- On Windows, a process that tries to rename while another is renaming, will get an "Access is denied".
- 04b5e47 includes a fix for this, where in Windows the rename is retried up to 8 times with progressively longer delays, if this error happens.
- This should cover even 8 different dpm processes trying to concurrently write to the same .dar file. Having different dpm processes trying to write to the same .dar file concurrently would be weird enough. Having more than 8 of them would be even weirder, so we consider this fix to be sensible. Happy to discuss other options if you have some pushback.
UNRELEASED.md: included description in d8ccfe1.
There seem to be some issues right now with GH Actions so the Windows tests CI is queued but not running in our fork. If you prefer, I'll let you know once that's resolved and I have verified that tests pass in Windows too. I'll run it a few times to account for race conditions. Windows tests passed several times on our fork's CI.
Open issues from last Copilot review addressed.

ffarall · 2026-10-06T13:17:55Z

@dasormeter Update. Windows tests CI has run successfully a few times in a row on our fork's CI.

ffarall · 2026-10-06T13:37:33Z

The Windows tests CI were failing in go tidy because of a duplicated require for the flock dependency. That's why I had to change go.mod again and it's triggering the fail in the Static checks / workflow-policy CI.

dasormeter · 2026-10-06T13:48:02Z

 	oras.land/oras-go/v2 v2.6.2
 )

-require github.com/gofrs/flock v0.13.1


you should be able to remove this change now as it is already included as a dependency

No changes left in go.mod 👌🏼

Windows tests and Static checks CIs passing in fork.

feat: add support for git dependencies

b2033a5

hrischuk-da reviewed Aug 21, 2026

View reviewed changes

ffarall mentioned this pull request Aug 25, 2026

Git-Based DAR Dependencies for dpm #105 - Milestone 1: Design, Implementation, and Upstream PR canton-foundation/canton-dev-fund#666

Open

gonzamontiel mentioned this pull request Aug 26, 2026

docs: update Git Dar dependencies docs canton-network/cf-docs#1463

Open

sammy-da reviewed Aug 27, 2026

View reviewed changes

gonzamontiel and others added 4 commits August 31, 2026 10:04

refactor: group git dependency fields into a its own struct

0ef4cd6

refactor: extract git dependency parsing into pkg/gitparse

773d773

fix: drop structured yaml feature

7ae6aa6

Merge branch 'main' into proposal/git-dependencies-support

de75365

Merge branch 'main' into proposal/git-dependencies-support

72a1eae

gonzamontiel requested review from dasormeter, hrischuk-da and sammy-da September 30, 2026 13:48

Merge remote-tracking branch 'fork/main' into proposal/git-dependenci…

cbb1516

…es-support

dasormeter requested a balanced review from Copilot October 1, 2026 20:09

Copilot started reviewing on behalf of dasormeter October 1, 2026 20:10 View session

This comment was marked as outdated.

Sign in to view

ffarall added 2 commits October 2, 2026 14:33

Merge branch 'main' into proposal/git-dependencies-support

e0ef265

fix: 🐛 Lock directory where repository is being pulled

6d7c138

dasormeter reviewed Oct 2, 2026

View reviewed changes

Comment thread go.mod Outdated

ffarall added 3 commits October 4, 2026 19:24

Merge remote-tracking branch 'upstream/main' into proposal/git-depend…

0c24cdb

…encies-support

test: 🐛 Fix tests for windows

53d3483

test: 🐛 Fix tests for windows

1296f9f

fix: 🐛 Create unique temporary files to avoid concurrency problems

0b73c56

dasormeter mentioned this pull request Oct 5, 2026

add flock dependency #323

Merged

dasormeter requested review from dasormeter and a balanced review from Copilot October 5, 2026 13:16

Copilot started reviewing on behalf of dasormeter October 5, 2026 13:17 View session

Copilot AI reviewed Oct 5, 2026

View reviewed changes

Comment thread pkg/gitparse/format.go

Comment thread pkg/packagelock/locker_git.go Outdated

Comment thread pkg/githubrelease/githubrelease.go Outdated

ffarall added 2 commits October 5, 2026 13:26

fix: 🐛 Retry on "Access is denied" for Windows when renaming

04b5e47

docs: 📝 Update UNRELEASED.md with new git dependencies feature

d8ccfe1

ffarall added 4 commits October 5, 2026 16:19

fix: 🐛 Keep .git suffix and separate scheme based on ://

c383e29

Merge remote-tracking branch 'upstream/main' into proposal/git-depend…

64efdf2

…encies-support

fix: 🩹 Reject ambiguous git pin in lockfile

3e4d4eb

fix: 🩹 Reject empty release download

5cbf34a

test: 🩹 Remove duplicate require for flock

93d1205

dasormeter reviewed Oct 6, 2026

View reviewed changes

revert: ⏪ Remove changes in go.mod

7f3992b

Conversation

gonzamontiel commented Aug 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

Summary

Technical design

Git dependencies core

Lock + update

GitHub releases

UX extras

Coverage

Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sammy-da Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sammy-da Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sammy-da Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tomimor commented Sep 21, 2026

Uh oh!

This comment was marked as outdated.

Uh oh!

Uh oh!

dasormeter commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ffarall commented Oct 2, 2026

Uh oh!

dasormeter commented Oct 2, 2026

Uh oh!

dasormeter commented Oct 2, 2026

Uh oh!

dasormeter commented Oct 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dasormeter commented Oct 5, 2026

Uh oh!

ffarall commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ffarall commented Oct 6, 2026

Uh oh!

ffarall commented Oct 6, 2026

Uh oh!

gonzamontiel commented Aug 21, 2026 •

edited

Loading

sammy-da Aug 27, 2026 •

edited

Loading

sammy-da Aug 27, 2026 •

edited

Loading

sammy-da Aug 27, 2026 •

edited

Loading

dasormeter commented Oct 2, 2026 •

edited

Loading

ffarall commented Oct 5, 2026 •

edited

Loading