Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
57 commits
Select commit Hold shift + click to select a range
8053aa5
early mods of the new export data provider framework #11405
landreev Jun 5, 2025
ebbde8b
... and had to change it back temporarily
landreev Jun 5, 2025
dcae3fc
re-checking in the updated data provider interface. #11405
landreev Jun 12, 2025
788dc4d
Merge branch 'develop' into 11405-refactor-exports
landreev Aug 25, 2025
be3603c
intermediate state of the export data provider implementation using t…
landreev Aug 26, 2025
90757da
removed some experimental code. #11405
landreev Aug 26, 2025
fbd4998
work in progress
landreev Aug 27, 2025
3cdb98f
work in progress (this is working for me, since I have dataverse-spi …
landreev Sep 2, 2025
de62287
alternative, batch implementation #11405
landreev Sep 2, 2025
4fd6a0e
logging statements for debugging #11405
landreev Sep 4, 2025
a8a7ac9
A quick fix for the batch-based datatable metadata processing #11405
landreev Sep 5, 2025
b1ed132
implemented a dedicated method in the VersionFileService for looking up
landreev Sep 8, 2025
a769133
Merge branch 'develop' into 11405-refactor-exports
landreev Sep 8, 2025
933f6a7
experimental attempts to build with snapshot releases of dataverse-sp…
landreev Sep 9, 2025
a5ca53d
fixing the typo in the repo url :( #11405
landreev Sep 9, 2025
11cd77c
Adding the option of re-exporting select formats only to the API #11405
landreev Sep 16, 2025
f9751e1
Merge branch 'develop' into 11405-refactor-exports
landreev Nov 5, 2025
7dac667
Made the batch-sizing math smarter (processing datavariable metadata …
landreev Nov 10, 2025
b2dc80e
first stab at making the internal exportdataprovider use the latest i…
landreev May 6, 2026
da772b4
some cleanup/debugging needed to get the exports working again, when …
landreev May 8, 2026
db203c6
extra data consistency logic #11405
landreev May 26, 2026
515e0ea
Added further data consistency checks to avoid any ambiguity in how t…
landreev May 27, 2026
43951fe
Merge branch 'develop' into 11405-refactor-exports-experiment
landreev Jun 18, 2026
cd1a9ed
code changes that were needed in develop in order to sync up the expo…
landreev Jun 18, 2026
22b2d39
New tests added, some cleanup (this is still in the "experiment" sub-…
landreev Jul 20, 2026
e096244
Merge pull request #12546 from IQSS/11405-refactor-exports-experiment
landreev Jul 20, 2026
523bf0e
Minor fix in Oliver's sample JpaEntityManageService test. It was work…
landreev Jul 20, 2026
4662b9f
Adding the new, JpaEntityManagerService-based test to the list of reg…
landreev Jul 20, 2026
1eb8420
Merge branch 'develop' into 11405-refactor-exports
landreev Jul 20, 2026
f24c726
Removed modules/dataverse-spi and the action that used to build snaps…
landreev Jul 21, 2026
3d318be
... failed to add this change to the commit; unit tests have just fai…
landreev Jul 21, 2026
a8779dc
adding "performance" to it.groups in pom.xml #11405
landreev Jul 22, 2026
cc006e2
Nuking an old, unused method "kept for reference" for some time now. …
landreev Jul 22, 2026
a5d8a37
removed an unnecessary C-style variable definition. #11405
landreev Jul 22, 2026
7a476cf
Replaced a direct injection of an EJB with a constructor in the data …
landreev Jul 23, 2026
d119c22
Minor refactoring per review #11405
landreev Jul 24, 2026
d934f85
Minor refactoring per review #11405
landreev Jul 24, 2026
1d9a4d3
Cached JsonArray was cached between 2 semantically different methods …
landreev Jul 24, 2026
e0e73a8
Switched to a secure DocumentBuilderFactory #11405
landreev Jul 24, 2026
a8e808d
A better/lazier way of returning a Stream of the retrieved objects. #…
landreev Jul 28, 2026
b34309c
Removed an unnecessary for() loop #11405
landreev Jul 28, 2026
e2214a9
Streaming refactored in the non-paginated new getDatasetFileDetails()…
landreev Jul 28, 2026
9f0e0cb
Added support for other search predicates, including non-tabular file…
landreev Jul 28, 2026
bdbaeff
sort order. #11405
landreev Jul 28, 2026
a52c8e9
refactored the branch to work with the latest 2.1.0 spi snapshot #11405
landreev Jul 31, 2026
522631d
some cleanup: validation of the format names on the API level, etc. #…
landreev Aug 8, 2026
4e449d3
Merge branch 'develop' into 11405-refactor-exports
landreev Aug 13, 2026
a215a3e
Resolved conflicts:
landreev Aug 19, 2026
9012cc9
switch to JsonUtil #11405
pdurbin Aug 19, 2026
1223fd2
Merge branch 'develop' into 11405-refactor-exports
landreev Aug 19, 2026
85707d1
Apply suggestion from @pdurbin
landreev Aug 21, 2026
dd1eeda
Apply suggestion from @pdurbin
landreev Aug 21, 2026
6e3bf78
cosmetic/deleting unused code #11405
landreev Aug 21, 2026
298da74
A few doc. additions #11405
landreev Aug 21, 2026
c2443d8
typos and such #11405
landreev Aug 21, 2026
43e2eb0
... more cosmetic stuff #11405
landreev Aug 21, 2026
5802628
No need to mention the new parameter in the changelog. #11405
landreev Aug 21, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
94 changes: 0 additions & 94 deletions .github/workflows/spi_release.yml

This file was deleted.

2 changes: 2 additions & 0 deletions doc/release-notes/11405-export-improvements.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
Dataverse metadata export code has been re-factored to use the new and improved Data Export framework (note also that the relevant interface and the accompanying code have been moved out of the main Dataverse repository and into the dedicated [GDCC project](https://github.com/gdcc/dataverse-spi)). In this release this will result in a measurable improvement in exporting the DDI format. But it will be possible to take advantage of this refactoring in the exports of other data-rich formats in future releases.
An optional query parameter ``formats`` has been added to the ``reExportAll`` and ``reExportDataset``, allowing an administrator to force re-export of only the formats specified.
Comment on lines +1 to +2

@pdurbin pdurbin Aug 21, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can wordsmith this at release time but a few thoughts.

I'd probably pitch this as two related features:

## Highlights

### Export improvements

- Faster DDI export
- Ability to export only selected formats 

Then, further down:

## API updates

### Ability to export only selected formats

curl example, link to API Guide.

Still further down:

## Updates for developers

We've refactored, blah blah blah.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, makes sense. Will do.

7 changes: 7 additions & 0 deletions doc/sphinx-guides/source/admin/metadataexport.rst
Original file line number Diff line number Diff line change
Expand Up @@ -24,13 +24,16 @@ In addition to the automated exports, a Dataverse installation admin can start a

``curl http://localhost:8080/api/admin/metadata/reExportAll?olderThan=<YYYY-MM-DD>``

``curl http://localhost:8080/api/admin/metadata/reExportAll?formats=Datacite,croissant``

``curl http://localhost:8080/api/admin/metadata/clearExportTimestamps``

``curl http://localhost:8080/api/admin/metadata/:persistentId/reExportDataset?persistentId=doi:10.5072/FK2/AAA000``

The first will attempt to export all the published, local (non-harvested) datasets that haven't been exported yet.
The second will *force* a re-export of every published, local dataset, regardless of whether it has already been exported or not.
With the optional olderThan query parameter, the second will *force* re-export of all published, local datasets that were last exported before the olderThan date.
Another optional parameter, ``formats`` (new as of v6.12) allows an administrator to force a reexport of specific metadata formats only.

The first two calls return a status message informing the administrator that the process has been launched (``{"status":"WORKFLOW_IN_PROGRESS"}``). The administrator can check the progress of the process via log files: ``[Payara directory]/glassfish/domains/domain1/logs/export_[time stamp].log``.

Expand All @@ -44,6 +47,10 @@ reExportDataset can be called with either ``persistentId`` (as shown above, with

``curl http://localhost:8080/api/admin/metadata/42/reExportDataset``

An optional parameter, ``formats`` (new as of v6.12) can be used to force a reexport of specific metadata formats only. For example:

``curl http://localhost:8080/api/admin/metadata/42/reExportDataset?formats=oai_dc,oai_ddi``

Note, that creating, modifying, or re-exporting an OAI set will also attempt to export all the unexported datasets found in the set.

Export Failures
Expand Down
4 changes: 1 addition & 3 deletions modules/dataverse-parent/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,6 @@
<module>../../pom.xml</module>
<module>../../scripts/zipdownload</module>
<module>../container-base</module>
<module>../dataverse-spi</module>
</modules>

<!-- Transitive dependencies, bigger library "bill of materials" (BOM) and
Expand Down Expand Up @@ -437,8 +436,6 @@
</snapshots>
</repository>
<!-- Uncomment when using snapshot releases from Maven Central -->
<!-- Note the new - as of Sept. 2025 sonatype url -->
<!--
<repository>
<id>central-portal-snapshots</id>
<name>Central Portal Snapshots</name>
Expand All @@ -452,6 +449,7 @@
<enabled>true</enabled>
</snapshots>
</repository>
<!--
<repository>
<id>s01-oss-sonatype</id>
<name>s01-oss-sonatype</name>
Expand Down
1 change: 0 additions & 1 deletion modules/dataverse-spi/.gitignore

This file was deleted.

242 changes: 0 additions & 242 deletions modules/dataverse-spi/pom.xml

This file was deleted.

Loading
Loading