Allow to use Gitlab registry as index + Allow pypi to be used for dependency resolution#82
Open
grische wants to merge 5 commits intopython-morgan:mainfrom
Open
Allow to use Gitlab registry as index + Allow pypi to be used for dependency resolution#82grische wants to merge 5 commits intopython-morgan:mainfrom
grische wants to merge 5 commits intopython-morgan:mainfrom
Conversation
Refactor the code such that it is easier to expand the code for alternative target repositories.
Contributor
Author
|
If you prefer, I can also split this PR into two if this helps reviewing. |
Also add tests for the new method.
b6a38fa to
3297da7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR contains two distinct features:
PyPI for dependency resolution
This feature stops extracting and parsing the files from the local mirror, but instead uses PyPi's json API.
It is slower than local file parsing, but a lot less error-prone as the PyPi package parsing is very mature and constantly tested and verified.
GitLab registry as index
This feature uses a Gitlab registry as a target registry instead of relying on a local file system structure.
It checks which packages are already present in the Gitlab registry and only fetches those that are still missing. All the missing packages can then be uploaded by tools like, e.g. twine after Morgan has finished mirroring.
As Gitlab doesn't have PyPi's json API to obtain package dependencies and fetching all files from the Gitlab registry is very expensive on larger registries, this feature only makes sense together with PyPi as a dependency resolution.
As this is a major feature, we also provide extensive tests for both the local-file registry as well as the Gitlab registry.