-
Notifications
You must be signed in to change notification settings - Fork 561
Expands InternalExportDataProvider per dataverse-spi 2.1.0 and refactors the DDI exporter to utilize the new functionality #11799
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
landreev
wants to merge
57
commits into
develop
Choose a base branch
from
11405-refactor-exports
base: develop
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+1,579
−989
Open
Changes from all commits
Commits
Show all changes
57 commits
Select commit
Hold shift + click to select a range
8053aa5
early mods of the new export data provider framework #11405
landreev ebbde8b
... and had to change it back temporarily
landreev dcae3fc
re-checking in the updated data provider interface. #11405
landreev 788dc4d
Merge branch 'develop' into 11405-refactor-exports
landreev be3603c
intermediate state of the export data provider implementation using t…
landreev 90757da
removed some experimental code. #11405
landreev fbd4998
work in progress
landreev 3cdb98f
work in progress (this is working for me, since I have dataverse-spi …
landreev de62287
alternative, batch implementation #11405
landreev 4fd6a0e
logging statements for debugging #11405
landreev a8a7ac9
A quick fix for the batch-based datatable metadata processing #11405
landreev b1ed132
implemented a dedicated method in the VersionFileService for looking up
landreev a769133
Merge branch 'develop' into 11405-refactor-exports
landreev 933f6a7
experimental attempts to build with snapshot releases of dataverse-sp…
landreev a5ca53d
fixing the typo in the repo url :( #11405
landreev 11cd77c
Adding the option of re-exporting select formats only to the API #11405
landreev f9751e1
Merge branch 'develop' into 11405-refactor-exports
landreev 7dac667
Made the batch-sizing math smarter (processing datavariable metadata …
landreev b2dc80e
first stab at making the internal exportdataprovider use the latest i…
landreev da772b4
some cleanup/debugging needed to get the exports working again, when …
landreev db203c6
extra data consistency logic #11405
landreev 515e0ea
Added further data consistency checks to avoid any ambiguity in how t…
landreev 43951fe
Merge branch 'develop' into 11405-refactor-exports-experiment
landreev cd1a9ed
code changes that were needed in develop in order to sync up the expo…
landreev 22b2d39
New tests added, some cleanup (this is still in the "experiment" sub-…
landreev e096244
Merge pull request #12546 from IQSS/11405-refactor-exports-experiment
landreev 523bf0e
Minor fix in Oliver's sample JpaEntityManageService test. It was work…
landreev 4662b9f
Adding the new, JpaEntityManagerService-based test to the list of reg…
landreev 1eb8420
Merge branch 'develop' into 11405-refactor-exports
landreev f24c726
Removed modules/dataverse-spi and the action that used to build snaps…
landreev 3d318be
... failed to add this change to the commit; unit tests have just fai…
landreev a8779dc
adding "performance" to it.groups in pom.xml #11405
landreev cc006e2
Nuking an old, unused method "kept for reference" for some time now. …
landreev a5d8a37
removed an unnecessary C-style variable definition. #11405
landreev 7a476cf
Replaced a direct injection of an EJB with a constructor in the data …
landreev d119c22
Minor refactoring per review #11405
landreev d934f85
Minor refactoring per review #11405
landreev 1d9a4d3
Cached JsonArray was cached between 2 semantically different methods …
landreev e0e73a8
Switched to a secure DocumentBuilderFactory #11405
landreev a8e808d
A better/lazier way of returning a Stream of the retrieved objects. #…
landreev b34309c
Removed an unnecessary for() loop #11405
landreev e2214a9
Streaming refactored in the non-paginated new getDatasetFileDetails()…
landreev 9f0e0cb
Added support for other search predicates, including non-tabular file…
landreev bdbaeff
sort order. #11405
landreev a52c8e9
refactored the branch to work with the latest 2.1.0 spi snapshot #11405
landreev 522631d
some cleanup: validation of the format names on the API level, etc. #…
landreev 4e449d3
Merge branch 'develop' into 11405-refactor-exports
landreev a215a3e
Resolved conflicts:
landreev 9012cc9
switch to JsonUtil #11405
pdurbin 1223fd2
Merge branch 'develop' into 11405-refactor-exports
landreev 85707d1
Apply suggestion from @pdurbin
landreev dd1eeda
Apply suggestion from @pdurbin
landreev 6e3bf78
cosmetic/deleting unused code #11405
landreev 298da74
A few doc. additions #11405
landreev c2443d8
typos and such #11405
landreev 43e2eb0
... more cosmetic stuff #11405
landreev 5802628
No need to mention the new parameter in the changelog. #11405
landreev File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file was deleted.
Oops, something went wrong.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| Dataverse metadata export code has been re-factored to use the new and improved Data Export framework (note also that the relevant interface and the accompanying code have been moved out of the main Dataverse repository and into the dedicated [GDCC project](https://github.com/gdcc/dataverse-spi)). In this release this will result in a measurable improvement in exporting the DDI format. But it will be possible to take advantage of this refactoring in the exports of other data-rich formats in future releases. | ||
| An optional query parameter ``formats`` has been added to the ``reExportAll`` and ``reExportDataset``, allowing an administrator to force re-export of only the formats specified. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We can wordsmith this at release time but a few thoughts.
I'd probably pitch this as two related features:
Then, further down:
Still further down:
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure, makes sense. Will do.