From 619351ef3a4c08159eba7ab11e7790975fed0a2d Mon Sep 17 00:00:00 2001 From: costero-e Date: Wed, 10 Jun 2026 14:52:27 +0200 Subject: [PATCH 01/10] adding new SOP0014 add datasets medatada to node beacon --- ...14_add-datasets-metadata-to-node-beacon.md | 283 ++++++++++++++++++ 1 file changed, 283 insertions(+) create mode 100644 sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md diff --git a/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md b/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md new file mode 100644 index 0000000..7570475 --- /dev/null +++ b/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md @@ -0,0 +1,283 @@ +# European GDI - Add dataset’s metadata to Node beacon + +| Metadata | Value | +| -------------------- | --------------------------------------------------------------------- | +| Template SOP number | `GDI-SOP0014` | +| Template SOP version | `v1` | +| Topic | Technical infrastructure & software development | +| Template SOP Type | Node-specific SOP | +| GDI Node | | +| Instance version | | + +## Index + +1. [Document History](#1-document-history) +2. [Glossary](#2-glossary) +3. [Roles and Responsibilities](#3-roles-and-responsibilities) +4. [Purpose](#4-purpose) +5. [Scope](#5-scope) +6. [Introduction and Background Information](#6-introduction-and-background-information) +7. [Summary or Context Diagram](#7-summary-or-context-diagram) +8. [Procedure](#8-procedure) +9. [References](#9-references) + +### 1. Document History + +| Template Version | Instance version | Author(s) | Description of changes | Date | +| ---------------- | ---------------- | --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------ | +| `v1` | | Liina Nagirnaja, Oriol López-Doriga Sagalés | First version of the SOP for GitHub issue [#66](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/issues/66), after reviews completed. | `2026.06.09` | +| `v0` | | Liina Nagirnaja, Oriol López-Doriga Sagalés | Initial version of the SOP for GitHub issue [#66](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/issues/66), based on the approved draft and reviewed copy. | `2026.04.30` | + +### 2. Glossary + +Find GDI SOPs common Glossary at the [**charter document**](../../docs/GDI-SOP_charter.md). + +| Abbreviation | Description | +| ------------- | ------------------------------------------------- | +| AF | Allele Frequency | +| CRG | Centre for Genomic Regulation | +| DKFZ | Deutsches Krebsforschungszentrum | +| GDI | European Genomic Data Infrastructure | +| GDPR | General Data Protection Regulation. | +| EBI | European Bioinformatics Institute | +| EMBL | European Molecular Biology Laboratory | +| FAIR | Findability, Accessibility, Interoperability and Reusability | +| HTTPS | Hypertext Transfer Protocol Secure | +| ID | Identifier | +| SOP | Standard Operating Procedure | +| UM | University of Maribor | +| VHD | Virtual Helpdesk | + +| Term | Definition | +| ------------- | ----------------------------------------------------------------------------------------------------------| | +| Beacon query | HTTPS request to an endpoint of the node’s Allele Frequency beacon. | +| Dataset permissions | All the information related to the dataset grants and its security level configuration. | + +### 3. Roles and Responsibilities + +See qualifications and responsibilities of the roles at the [**Organisational Roles and Responsibilities**](../../docs/GDI-SOP_organisational-roles-and-responsibilities.md) document. + +| Role | Full name | GDI/node role | Organisation | +| ---------- | --------------------------- | ------------------------------------------------ | ------------------------------------- | +| Author | Liina Nagirnaja | Beacon Manager | CRG | +| Author | Jordi Rambla | Beacon Product Owner | CRG | +| Author | Oriol López-Doriga Sagalés | Beacon Developer | CRG | +| Reviewer | Wasiu Akanni | Task 4.3 member | DKFZ | +| Reviewer | Aleš Čep | Task 4.3 member | UM | +| Reviewer | Marcos Casado Barbero | Task 4.3 member | EMBL-EBI | +| Approver | Gabriele Rinck | Task 4.3 member | EMBL-EBI | +| Authorizer | Management Board | Authorizer according to GDI SOP governance | GDI | + +### 4. Purpose + +Data loaded into GDI, be it through beacon or through FAIR Data Points (FDP), need to be consistent and queryable. In order to do so, validation of the data prior to submission needs to be performed. + +### 5. Scope + +The SOP covers node-level guidance for uploading dataset metadata to beacon and includes: +- Dataset metadata preparation and check that the dataset ID matches FDP requirements (GH) +- Entering dataset metadata to beacon (dataset ID and name) +- Validating the dataset endpoint with Verifier + +Out of scope for this SOP are: +- Full GDI HDM entry or validation +- Uploading variant records to beacon. For MAP1, the specific requirements are covered in the beacon guidelines documentation (here and here). +- Broader validation across services + +### 6. Introduction and Background Information + +Dataset metadata provides the foundation for discovering related data across the different GDI platforms and databases. When supplying the dataset metadata for a beacon, a kind of a linkage or contract between the boundaries of the beacon and the rest of the GDI components is being set. If the dataset metadata is wrong, these connections across all GDI components will fail, hence the importance of having a SOP for each component, in this case for beacon, that reduces the chances of this information to be misloaded. +For a broader context of GDI SOPs, please refer to the [Charter](../../docs/GDI-SOP_charter.md#4-introduction). + +### 7. Summary or Context Diagram + +```mermaid + A["Generate metadata file with the dataset's information"] --> B{"Confirm the type of operation to perform"} + A -- Failure --> D["Record failure details and report to VHD"] + B -- Insert new --> C["Insert metadata for new dataset"] + B -- Update existing --> E["Update metadata for existing dataset"] + C -- Failure --> D["Record failure details and report to VHD"] + E -- Failure --> D["Record failure details and report to VHD"] + C -- Success --> F["Add permissions for your new dataset"] + F -- Success --> I["Make additional statements for your dataset"] + E -- Success --> G["Validate your new dataset's metadata record with verifier, log and report completion"] + F -- Success --> G["Validate your new dataset's metadata record with verifier, log and report completion"] + G -- Valid --> H["Record the action in the local audit or change log, including who performed the change, when it was performed, what was changed and the approved scope"] + G -- Invalid --> D["Record failure details and report to VHD"] + I --> G["Validate your new dataset's metadata record with verifier, log and report completion"] +``` + +### 8. Procedure + +#### 8.1. Generate metadata file with the dataset’s information + +| Step identifier | When | Who | +| :-------------- | :------------------------------------------------------------------ | :-------------------------------------- | +| `1` | When metadata for a new beacon dataset needs to be added or existing metadata needs to be updated. | Node beacon maintainer | + +After collecting all the information related to the dataset’s metadata, create a new `datasets.csv` file with this exact name inside `/ri-tools/csv` folder (or any subfolder). Copy the headers you need from the template file `/ri-tools/csv/templates/datasets.csv`. Make sure the following mandatory headers are included: +```yaml +id +name +``` +Fill in the metadata in the row after the header. Ensure that the id follows the required [link](https://raw.githubusercontent.com/GenomicDataInfrastructure/gdi-metadata/refs/heads/main/Formulasation(shacl)/core/PiecesShape/Dataset.ttl) FDP identifier format and check that it is spelled exactly as the FDP dataset identifier for the same dataset. +Next step is to tell the ri-tools tool where the metadata file is stored. In `/ri-tools/conf/conf.py` update the value of the `csv_folder` configuration variable so that it points to the folder containing the dataset metadata file, datasets.csv. Example: +```python +csv_folder = './csv/' +``` +The folder may be any folder you create for this purpose, but it must contain only the relevant datasets.csv file for the dataset being uploaded. Do not include metadata files from other datasets or templates in the same folder. +Once done, execute next the script to generate a JSON file with the metadata for your dataset from the CSV you just created: +```bash +docker exec -it ri-tools python csv_to_bff.py +``` +- If the file has been successfully created, proceed to ⏩[Step 2](#82-confirm-the-type-of-operation-to-perform). +- If you encounter any issues, record the response obtained from the used commands, adding all the information about the actions performed and the intended goal of performing them and report to the GDI Virtual Helpdesk so that requester communication continues through the VHD workflow. + + +#### 8.2. Confirm the type of operation to perform + +| Step identifier | When | Who | +| :-------------- | :------------------------------------------------------------------ | :-------------------------------------- | +| `2` | After successfully completing ⏩[Step 1](#81-generate-metadata-file-with-the-datasets-information). | Node beacon maintainer | + +As the node beacon maintainer, confirm that the incoming request package is complete before performing any modification in beacon. Double check first: +- What dataset’s metadata needs to be uploaded +- If the dataset was already uploaded in the beacon +Send an HTTPS GET request to your beacon’s datasets endpoint and locate the record corresponding to the ID of the dataset regarding the metadata to be added, use the method you prefer (e.g., curl, postman...)...", +```bash +curl 'https:///api/datasets' +``` +or perform the dataset ID beacon query directly, +```bash +curl 'https:///api/datasets/' +``` +- If the dataset ID to upload metadata for was not found, you will need to perform an insert, proceed to ⏩[Step 3.1](#831-insert-metadata-for-new-dataset) +- If the dataset ID to upload metadata for was found, proceed to ⏩[Step 3.2](#832-update-metadata-for-existing-dataset) + + +##### 8.3. Adding dataset's metadata into beacon + +##### 8.3.1. Insert metadata for new dataset + +| Step identifier | When | Who | +| :-------------- | :--------------------------------------- | :-------------------------------------- | +| `3.1` | After confirming that the type of operation to perform is to insert metadata for a new dataset. | Node beacon maintainer | + +Insert the dataset’s metadata file into beacon by executing the following commands: +```bash +docker cp ri-tools/output_docs/datasets.json mongoprod:tmp/datasets.json + +bash -c ' +read -s -p "Mongo user: " MONGO_USER; echo +read -s -p "Mongo password: " MONGO_PASS; echo + +docker exec -i mongoprod < { + const { id, _id, ...fieldsToUpdate } = doc; + + + const res = db.datasets.updateOne( + { id }, + { \$set: fieldsToUpdate } + ); + + + printjson(res); +}); +" +' +``` +You will be asked to enter the mongodb credentials on the terminal prompt. +- If the update was successful, proceed to ⏩[Step 5](#85-make-additional-statements-for-your-dataset). +- If the update was not successful, record the response obtained from the used commands, adding all the information about the actions performed and the intended goal of performing them and report to the GDI Virtual Helpdesk so that requester communication continues through the VHD workflow. + +#### 8.4. Add permissions for your new dataset + +| Step identifier | When | Who | +| :-------------- | :--------------------------------------- | :-------------------------------------- | +| `4` | After successfully inserting new dataset metadata ⏩[Step 3.1](#831-insert-metadata-for-new-dataset). | Node beacon maintainer | + +In beacon container, add the dataset ID to the file in path `/beacon/permissions/datasets/datasets_permissions.yml` and add a new item under it with the security level as the first property, and default_entry_types_granularity as the mandatory property under the security level: +```yaml + + public: + default_entry_types_granularity: record +``` +This is only an example, set the security level as it is meant for the dataset and add further restrictions as needed. More information about security levels and granularity types can be found here: [link](https://github.com/EGA-archive/beacon2-pi-api/tree/main#making-a-dataset-publicregisteredcontrolled). +Verify that your dataset appears correctly by sending an HTTPS GET request to your beacon’s datasets endpoint and locate the record corresponding to the ID of the dataset to be deleted, use the method you prefer (e.g., curl, postman...)...", +```bash +curl 'https:///datasets' +``` +or perform the ID beacon query directly, +```bash +curl 'https:///api/datasets/' +``` +- If the dataset appears and you wish it to declare either if it is test mode, synthetic or deprecated, proceed to ⏩[Step 5](#85-make-additional-statements-for-your-dataset). +- If the dataset appears, and you don’t wish to make additional declarations for your dataset, proceed to ⏩[Step 6](#86-validate-your-new-datasets-metadata-record-with-verifier-log-and-report-completion). +- If the dataset is not found, record the response obtained from the used commands, adding all the information about the actions performed and the intended goal of performing them and report to the GDI Virtual Helpdesk so that requester communication continues through the VHD workflow. + +#### 8.5. Make additional statements for your dataset + +| Step identifier | When | Who | +| :-------------- | :--------------------------------------- | :-------------------------------------- | +| `4` | After successfully updating new dataset metadata ⏩[Step 3.2](#832-update-metadata-for-existing-dataset), after adding permissions for the new dataset ⏩[Step 4](#84-add-permissions-for-your-new-dataset) or after making additional declarations for your dataset ⏩[Step 5](#85-make-additional-statements-for-your-dataset). | Node beacon maintainer | + +In case you want your dataset to be declared as meant for test mode, specify its nature or deprecate it, you can by editing the `/beacon/conf/datasets/datasets_conf.yml` file. Add a new entry with the dataset id as the main property and add whatever three optional following items you want to declare for the dataset, setting them as `True`. +```yaml + + isTest: True + isSynthetic: True + isDeprecated: True +``` +After that, proceed to ⏩[Step 6](#86-validate-your-new-datasets-metadata-record-with-verifier-log-and-report-completion). + + +#### 8.6. Validate your new dataset’s metadata record with verifier, log and report completion + +| Step identifier | When | Who | +| :-------------- | :--------------------------------------- | :-------------------------------------- | +| `6` | After successfully updating new dataset metadata ⏩[Step 3.2](#832-update-metadata-for-existing-dataset), after adding permissions for the new dataset ⏩[Step 4](#84-add-permissions-for-your-new-dataset) or after making additional declarations for your dataset ⏩[Step 5](#85-make-additional-statements-for-your-dataset). | Node beacon maintainer | + +As the node beacon maintainer, proceed to validate your new dataset metadata by running the [beacon verifier](https://beacon-verifier-demo.ega-archive.org/) on your beacon instance. +Focus on the `/datasets` endpoint. +- If the endpoint is valid, record the action in the local audit or change log, including who performed the change, when it was performed, what was changed and the approved scope. +- If the endpoint is not valid, record the response obtained from the used commands, adding all the information about the actions performed and the intended goal of performing them and report to the GDI Virtual Helpdesk so that requester communication continues through the VHD workflow. + +### 9. References + +| Reference | Description | +| --------- | ----------- | +| [1](../../docs/GDI-SOP_charter.md) | European GDI - SOP Charter (including Glossary) | +| [2](../../docs/GDI-SOP_information-service-management.md) | European GDI - Procedures for Information Service Management for SOPs | +| [3](../../docs/GDI-SOP_organisational-roles-and-responsibilities.md) | European GDI - Organisational Roles and Responsibilities | +| [4](https://github.com/EGA-archive/beacon2-ri-tools-v2/tree/main) | EGA - Beacon v2 RI Tools v2 | +| [5](https://github.com/EGA-archive/beacon2-pi-api/tree/main) | EGA - Beacon v2 Production Implementation | From eaf56c0fd9dddf6de6dd424452924db3cd2778b6 Mon Sep 17 00:00:00 2001 From: costero-e Date: Wed, 10 Jun 2026 14:56:58 +0200 Subject: [PATCH 02/10] modifying chapter, changelog and readme --- CHANGELOG.md | 1 + docs/GDI-SOP_charter.md | 11 +++++++++-- sops/README.md | 3 ++- 3 files changed, 12 insertions(+), 3 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ba3ed4e..ad3c077 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - [``GDI-SOP0008_node-helpdesk-ticket-classification.md``](sops/node-specific/GDI-SOP0008_node-helpdesk-ticket-classification.md) - GDI Node Helpdesk Ticket Classification SOP. - [``GDI-SOP0009_dataset-withdrawal.md``](sops/european-level/GDI-SOP0009_dataset-withdrawal.md) - Withdrawal request GDI SOP. - [``.gitignore``](.gitignore) - File to ignore certain files in git commits. +- [``GDI-SOP0014_add-datasets-metadata-to-node-beacon.md``](sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md) - Add dataset’s metadata to Node beacon. ### Modified - [``GDI-SOP_github-management.md``](docs/GDI-SOP_github-management.md) - Added reference to recorded session diff --git a/docs/GDI-SOP_charter.md b/docs/GDI-SOP_charter.md index 2c086a2..0fddcda 100644 --- a/docs/GDI-SOP_charter.md +++ b/docs/GDI-SOP_charter.md @@ -25,14 +25,17 @@ The following table defines the abbreviations and terms relevant to GDI SOPs. | 1+MG DAC | 1+MG Data Access Committee | | AAI | Authentication and Authorization Infrastructure | | AE | Associated Entity | +| AF | Allele Frequency | | AI | Action Item | | AF | Application Form | | API | Application Programming Interface | | BEN | Beneficiary | | CC | Carbon Copy (used in communications to send a copy to others) | +| CRG | Centre for Genomic Regulation | | CSC | Finnish IT Center for Science | | DAA | Data Access Agreement | | DAC | Data Access Committee | +| DKFZ | Deutsches Krebsforschungszentrum | | DMP | Data Management Plan | | DO | Data Object | | DP | Data Protection | @@ -56,7 +59,8 @@ The following table defines the abbreviations and terms relevant to GDI SOPs. | GH | GitHub | | HD | Helpdesk | | HRI | Health Research Infrastructure | -| ID | Identity Document | +| HTTPS | Hypertext Transfer Protocol Secure | +| ID | Identifier | | IdP | Identity Provider | | INSERM | L'Institut national de la santé et de la recherche médicale (France) | | ISM | Information Service Management | @@ -97,6 +101,7 @@ The following table defines the abbreviations and terms relevant to GDI SOPs. | UX | User experience | | VHD | Virtual Helpdesk | | VPN | Virtual Private Network | +| UM | University of Maribor | | UP | User portal | | WG | Working Group | | WP | Work Package | @@ -107,11 +112,13 @@ The following table defines the abbreviations and terms relevant to GDI SOPs. |------------------|-------------------------------------------------------------------------| | API | Application Programming Interface is a connection between computers or between computer programs. It is a type of software interface, offering a service to other pieces of software. | | Beacon |A web-accessible service implementing the GA4GH Beacon specification, enabling federated discovery (and sometimes retrieval) of genomic variant and related biomedical data across distributed resources| +| Beacon Query | HTTPS request to an endpoint of the node’s Allele Frequency beacon. | | CHANGELOG | Log or record of all notable changes made to a project | | Data controller |The legal owner or organisation that determines the purposes and means of processing the dataset and can authorise its withdrawal| +| Dataset permissions | All the information related to the dataset grants and its security level configuration. | | Dataset version |A specific released state of a dataset identified by a version label/number, used to distinguish minor or major changes between releases| | FitSM | Standards for lightweight IT Services Management; developed initially through The FedSM Project, funded by the European Commission, now maintained by ITEMO () | -| Hard-deletion |Complete, irreversible erasure of the data from all storage media so it cannot be recovered| +| Hard-deletion | Complete, irreversible erasure of the data from all storage media so it cannot be recovered| | JIRA | Software product developed by Atlassian that allows bug tracking, issue tracking and agile project management. | | JSON | JavaScript Object Notation is an open standard file format and data interchange format that uses human-readable text to store and transmit data objects consisting of name–value pairs and arrays (or other serializable values).| | Node Helpdesk | The support team is responsible for handling tickets related to data management issues associated with the node in the GDI Project. | diff --git a/sops/README.md b/sops/README.md index 2f7b90a..042a439 100644 --- a/sops/README.md +++ b/sops/README.md @@ -7,4 +7,5 @@ Below you can find the full list of SOPs in this repository. | [GDI-SOP0003_1+mg-dac-recommendation-approval.md](./european-level/GDI-SOP0003_1+mg-dac-recommendation-approval.md) | GDI-SOP0003 | v1 | Data & metadata management | European-Level SOP | | | 10 | 2024.11.11 | | [GDI-SOP0007_sop-template-creation.md](./european-level/GDI-SOP0007_sop-template-creation.md) | GDI-SOP0007 | v1 | Helpdesk & operations | European-Level SOP | | | 7 | 2024.10.29 | | [GDI-SOP0008_node-helpdesk-ticket-classification.md](./node-specific/GDI-SOP0008_node-helpdesk-ticket-classification.md) | GDI-SOP0008 | v1.0.0 | Helpdesk & operations | Node-specific SOP | | | 4 | 2026.03.10 | -| [GDI-SOP0009_dataset-withdrawal.md](./european-level/GDI-SOP0009_dataset-withdrawal.md) | GDI-SOP0009 | v1 | Data & metadata management | European-level SOP | | | 9 | 2026.03.10 | \ No newline at end of file +| [GDI-SOP0009_dataset-withdrawal.md](./european-level/GDI-SOP0009_dataset-withdrawal.md) | GDI-SOP0009 | v1 | Data & metadata management | European-level SOP | | | 9 | 2026.03.10 | +| [GDI-SOP0014_add-datasets-metadata-to-node-beacon](./node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md) | GDI-SOP0014 | v1 | Technical infrastructure & software development | Node-specific SOP | | | 6 | 2026.06.09 | \ No newline at end of file From 76c2f4456f476d19d8838b368838405eb63b5263 Mon Sep 17 00:00:00 2001 From: costero-e Date: Wed, 10 Jun 2026 17:16:32 +0200 Subject: [PATCH 03/10] fixing errors --- sops/README.md | 2 +- ...GDI-SOP0014_add-datasets-metadata-to-node-beacon.md | 10 +++++++--- 2 files changed, 8 insertions(+), 4 deletions(-) diff --git a/sops/README.md b/sops/README.md index 042a439..135dfd8 100644 --- a/sops/README.md +++ b/sops/README.md @@ -8,4 +8,4 @@ Below you can find the full list of SOPs in this repository. | [GDI-SOP0007_sop-template-creation.md](./european-level/GDI-SOP0007_sop-template-creation.md) | GDI-SOP0007 | v1 | Helpdesk & operations | European-Level SOP | | | 7 | 2024.10.29 | | [GDI-SOP0008_node-helpdesk-ticket-classification.md](./node-specific/GDI-SOP0008_node-helpdesk-ticket-classification.md) | GDI-SOP0008 | v1.0.0 | Helpdesk & operations | Node-specific SOP | | | 4 | 2026.03.10 | | [GDI-SOP0009_dataset-withdrawal.md](./european-level/GDI-SOP0009_dataset-withdrawal.md) | GDI-SOP0009 | v1 | Data & metadata management | European-level SOP | | | 9 | 2026.03.10 | -| [GDI-SOP0014_add-datasets-metadata-to-node-beacon](./node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md) | GDI-SOP0014 | v1 | Technical infrastructure & software development | Node-specific SOP | | | 6 | 2026.06.09 | \ No newline at end of file +| [GDI-SOP0014_add-datasets-metadata-to-node-beacon.md](./node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md) | GDI-SOP0014 | v1 | Technical infrastructure & software development | Node-specific SOP | | | 6 | 2026.06.09 | \ No newline at end of file diff --git a/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md b/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md index 7570475..3afd8a9 100644 --- a/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md +++ b/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md @@ -1,4 +1,4 @@ -# European GDI - Add dataset’s metadata to Node beacon +# European GDI - Add datasets metadata to Node beacon | Metadata | Value | | -------------------- | --------------------------------------------------------------------- | @@ -155,7 +155,11 @@ curl 'https:///api/datasets/' - If the dataset ID to upload metadata for was found, proceed to ⏩[Step 3.2](#832-update-metadata-for-existing-dataset) -##### 8.3. Adding dataset's metadata into beacon +#### 8.3. Adding dataset's metadata into beacon + +| Step identifier | When | Who | +| :-------------- | :--------------------------------------- | :-------------------------------------- | +| `3` | After confirming the type of operation to perform. | Node beacon maintainer | ##### 8.3.1. Insert metadata for new dataset @@ -249,7 +253,7 @@ curl 'https:///api/datasets/' | Step identifier | When | Who | | :-------------- | :--------------------------------------- | :-------------------------------------- | -| `4` | After successfully updating new dataset metadata ⏩[Step 3.2](#832-update-metadata-for-existing-dataset), after adding permissions for the new dataset ⏩[Step 4](#84-add-permissions-for-your-new-dataset) or after making additional declarations for your dataset ⏩[Step 5](#85-make-additional-statements-for-your-dataset). | Node beacon maintainer | +| `5` | After successfully updating new dataset metadata ⏩[Step 3.2](#832-update-metadata-for-existing-dataset), after adding permissions for the new dataset ⏩[Step 4](#84-add-permissions-for-your-new-dataset) or after making additional declarations for your dataset ⏩[Step 5](#85-make-additional-statements-for-your-dataset). | Node beacon maintainer | In case you want your dataset to be declared as meant for test mode, specify its nature or deprecate it, you can by editing the `/beacon/conf/datasets/datasets_conf.yml` file. Add a new entry with the dataset id as the main property and add whatever three optional following items you want to declare for the dataset, setting them as `True`. ```yaml From 769cef3adcde1a530ddb6a327448d54763e4f6db Mon Sep 17 00:00:00 2001 From: costero-e Date: Wed, 10 Jun 2026 17:19:30 +0200 Subject: [PATCH 04/10] fixing errors for SOP0009 --- sops/european-level/GDI-SOP0009_dataset-withdrawal.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sops/european-level/GDI-SOP0009_dataset-withdrawal.md b/sops/european-level/GDI-SOP0009_dataset-withdrawal.md index e218176..786e55e 100644 --- a/sops/european-level/GDI-SOP0009_dataset-withdrawal.md +++ b/sops/european-level/GDI-SOP0009_dataset-withdrawal.md @@ -312,7 +312,7 @@ Request details: - Deadline requested: ACTIONS NEEDED [Within 5 working days]: -1. Please follow through with the dataset withdrawal request as per [Step 5](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/sops/european-level/GDI-SOP0009_dataset-withdrawal.md#85-assign-withdrawal-response-team). +1. Please follow through with the dataset withdrawal request as per [Step 5](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/dev/sops/european-level/GDI-SOP0009_dataset-withdrawal.md#85-assign-withdrawal-response-team). 2. Update the VHD ticket with assigned leads and expected completion timeline. Once withdrawal is confirmed, the VHD will resume control to verify, notify the requester, and close the ticket. @@ -385,10 +385,10 @@ You are appointed **Technical Lead** (TL) for the dataset-withdrawal request det ACTIONS NEEDED: 1. Acknowledge receipt of this email **within 3 working days**. 2. Determine whether any (meta)data of the dataset resides in the specified GDI system. -3. If present, remove it (within the given scope) as per system SOP (see [Step 7](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/sops/european-level/GDI-SOP0009_dataset-withdrawal.md#87-per-system-dataset-withdrawal)). +3. If present, remove it (within the given scope) as per system SOP (see [Step 7](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/dev/sops/european-level/GDI-SOP0009_dataset-withdrawal.md#87-per-system-dataset-withdrawal)). 4. Report dataset withdrawal status in the ticket or directly contacting GDI's VHD. -You can find more details about how these tasks have been fulfilled previously at [Step 7](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/main/sops/european-level/GDI-SOP0009_dataset-withdrawal.md#87-per-system-dataset-withdrawal). +You can find more details about how these tasks have been fulfilled previously at [Step 7](https://github.com/GenomicDataInfrastructure/standard-operating-procedures/blob/dev/sops/european-level/GDI-SOP0009_dataset-withdrawal.md#87-per-system-dataset-withdrawal). If you need additional context or support, contact the OC/SDPC (gdi-sdpc [at] elixir-europe.org, gdi-oc [at] elixir-europe.org) or the VHD Officer (). Thank you for your prompt attention. @@ -436,7 +436,7 @@ To accomplish these tasks, **follow the appropriate SOP** for your specified sys - Dataset Browser (_#! SOP TBD_). - Nodes' FAIR Data Points (FDP) (_#! SOP TBD, see https://github.com/GenomicDataInfrastructure/standard-operating-procedures/issues/63 #!_). - Allele Frequency Browser (_#! SOP TBD_). - - Nodes' Allele Frequency Beacons (_#! SOP TBD, see https://github.com/GenomicDataInfrastructure/standard-operating-procedures/issues/65 #!_). + - Nodes' Allele Frequency Beacons ([GDI-SOP0012_withdraw-dataset-from-node-allele-frequency-beacon.md](../node-specific/GDI-SOP0012_withdraw-dataset-from-node-allele-frequency-beacon.md)). - **Nodes' Secure Processing Environment** (SPE) (_#! SOP TBD_). Once you have completed the relevant withdrawal SOP for the specified GDI System: From 546d7df3604cdb2b4da5d8c47a9c8c79662b61b3 Mon Sep 17 00:00:00 2001 From: costero-e Date: Wed, 10 Jun 2026 17:21:43 +0200 Subject: [PATCH 05/10] fixing errors for SOP0009 involving AF beacon SOP0012 --- sops/european-level/GDI-SOP0009_dataset-withdrawal.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/sops/european-level/GDI-SOP0009_dataset-withdrawal.md b/sops/european-level/GDI-SOP0009_dataset-withdrawal.md index 786e55e..27a59af 100644 --- a/sops/european-level/GDI-SOP0009_dataset-withdrawal.md +++ b/sops/european-level/GDI-SOP0009_dataset-withdrawal.md @@ -435,8 +435,6 @@ To accomplish these tasks, **follow the appropriate SOP** for your specified sys - Nodes' Beacons (_#! SOP TBD, see https://github.com/GenomicDataInfrastructure/standard-operating-procedures/issues/64 #!_). - Dataset Browser (_#! SOP TBD_). - Nodes' FAIR Data Points (FDP) (_#! SOP TBD, see https://github.com/GenomicDataInfrastructure/standard-operating-procedures/issues/63 #!_). - - Allele Frequency Browser (_#! SOP TBD_). - - Nodes' Allele Frequency Beacons ([GDI-SOP0012_withdraw-dataset-from-node-allele-frequency-beacon.md](../node-specific/GDI-SOP0012_withdraw-dataset-from-node-allele-frequency-beacon.md)). - **Nodes' Secure Processing Environment** (SPE) (_#! SOP TBD_). Once you have completed the relevant withdrawal SOP for the specified GDI System: From 41d4cce8f016469f5c282638e5e81a70aee6a5f2 Mon Sep 17 00:00:00 2001 From: costero-e Date: Mon, 22 Jun 2026 11:39:45 +0200 Subject: [PATCH 06/10] fixing table and flowchart --- .../GDI-SOP0014_add-datasets-metadata-to-node-beacon.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md b/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md index 3afd8a9..effdbc5 100644 --- a/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md +++ b/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md @@ -49,7 +49,7 @@ Find GDI SOPs common Glossary at the [**charter document**](../../docs/GDI-SOP_c | VHD | Virtual Helpdesk | | Term | Definition | -| ------------- | ----------------------------------------------------------------------------------------------------------| | +| ------------- | ----------------------------------------------------------------------------------------------------------| | Beacon query | HTTPS request to an endpoint of the node’s Allele Frequency beacon. | | Dataset permissions | All the information related to the dataset grants and its security level configuration. | @@ -92,6 +92,7 @@ For a broader context of GDI SOPs, please refer to the [Charter](../../docs/GDI- ### 7. Summary or Context Diagram ```mermaid +flowchart TD; A["Generate metadata file with the dataset's information"] --> B{"Confirm the type of operation to perform"} A -- Failure --> D["Record failure details and report to VHD"] B -- Insert new --> C["Insert metadata for new dataset"] From 37d7d39c73add1f78164125952cc702ddc772056 Mon Sep 17 00:00:00 2001 From: costero-e Date: Mon, 22 Jun 2026 12:55:13 +0200 Subject: [PATCH 07/10] changing reference deleting for inserting in section 8.4 --- .../GDI-SOP0014_add-datasets-metadata-to-node-beacon.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md b/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md index effdbc5..0c82d81 100644 --- a/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md +++ b/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md @@ -238,7 +238,7 @@ In beacon container, add the dataset ID to the file in path `/beacon/permissions default_entry_types_granularity: record ``` This is only an example, set the security level as it is meant for the dataset and add further restrictions as needed. More information about security levels and granularity types can be found here: [link](https://github.com/EGA-archive/beacon2-pi-api/tree/main#making-a-dataset-publicregisteredcontrolled). -Verify that your dataset appears correctly by sending an HTTPS GET request to your beacon’s datasets endpoint and locate the record corresponding to the ID of the dataset to be deleted, use the method you prefer (e.g., curl, postman...)...", +Verify that your dataset appears correctly by sending an HTTPS GET request to your beacon’s datasets endpoint and locate the record corresponding to the ID of the dataset to be inserted, use the method you prefer (e.g., curl, postman...)...", ```bash curl 'https:///datasets' ``` From 06f24db29f52ae980d723865d470a40a7e00d189 Mon Sep 17 00:00:00 2001 From: costero-e Date: Fri, 26 Jun 2026 12:11:09 +0200 Subject: [PATCH 08/10] some rewording, adding missing links and extra step for verifier --- ...-SOP0014_add-datasets-metadata-to-node-beacon.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md b/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md index 0c82d81..54803f3 100644 --- a/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md +++ b/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md @@ -42,6 +42,7 @@ Find GDI SOPs common Glossary at the [**charter document**](../../docs/GDI-SOP_c | EBI | European Bioinformatics Institute | | EMBL | European Molecular Biology Laboratory | | FAIR | Findability, Accessibility, Interoperability and Reusability | +| HD | Helpdesk | | HTTPS | Hypertext Transfer Protocol Secure | | ID | Identifier | | SOP | Standard Operating Procedure | @@ -70,7 +71,7 @@ See qualifications and responsibilities of the roles at the [**Organisational Ro ### 4. Purpose -Data loaded into GDI, be it through beacon or through FAIR Data Points (FDP), need to be consistent and queryable. In order to do so, validation of the data prior to submission needs to be performed. +Data loaded into GDI, be it through beacon or through FAIR Data Points (FDP), need to be consistent and queryable. This SOP aims to clarify how to proceed when new dataset metadata needs to be added to a Node's beacon, covering the cases for updating or inserting datasets and full validation of the incoming metadata. ### 5. Scope @@ -80,8 +81,8 @@ The SOP covers node-level guidance for uploading dataset metadata to beacon and - Validating the dataset endpoint with Verifier Out of scope for this SOP are: -- Full GDI HDM entry or validation -- Uploading variant records to beacon. For MAP1, the specific requirements are covered in the beacon guidelines documentation (here and here). +- Full GDI HD management entry or validation +- Uploading variant records to beacon. For MAP1, the specific requirements are covered in the beacon guidelines documentation ([here](https://docs.google.com/document/d/1LLzp6zZT3fSM1XxOXHuRqwJje1v726Z2/edit?rtpof=true&tab=t.0) and [here](https://docs.google.com/document/d/1rc0591dFHNghAYqv3SE6pNkfFmGrQgWAemmR1Iek3ss/edit?tab=t.0)). - Broader validation across services ### 6. Introduction and Background Information @@ -121,7 +122,7 @@ After collecting all the information related to the dataset’s metadata, create id name ``` -Fill in the metadata in the row after the header. Ensure that the id follows the required [link](https://raw.githubusercontent.com/GenomicDataInfrastructure/gdi-metadata/refs/heads/main/Formulasation(shacl)/core/PiecesShape/Dataset.ttl) FDP identifier format and check that it is spelled exactly as the FDP dataset identifier for the same dataset. +Fill in the metadata in the row after the header. Ensure that the id follows the required [Dataset SHACL](https://raw.githubusercontent.com/GenomicDataInfrastructure/gdi-metadata/refs/heads/main/Formulasation(shacl)/core/PiecesShape/Dataset.ttl), which needs to be consistent with the SHACL that the Node is currently using in their FDP and doesn't need to be the latest version from the gdi metadata repository. Next step is to tell the ri-tools tool where the metadata file is stored. In `/ri-tools/conf/conf.py` update the value of the `csv_folder` configuration variable so that it points to the folder containing the dataset metadata file, datasets.csv. Example: ```python csv_folder = './csv/' @@ -237,7 +238,7 @@ In beacon container, add the dataset ID to the file in path `/beacon/permissions public: default_entry_types_granularity: record ``` -This is only an example, set the security level as it is meant for the dataset and add further restrictions as needed. More information about security levels and granularity types can be found here: [link](https://github.com/EGA-archive/beacon2-pi-api/tree/main#making-a-dataset-publicregisteredcontrolled). +This is only an example, set the security level as it is meant for the dataset and add further restrictions as needed. More information about security levels and granularity types can be found in the [beacon2 pi repository"](https://github.com/EGA-archive/beacon2-pi-api/tree/main#making-a-dataset-publicregisteredcontrolled). Verify that your dataset appears correctly by sending an HTTPS GET request to your beacon’s datasets endpoint and locate the record corresponding to the ID of the dataset to be inserted, use the method you prefer (e.g., curl, postman...)...", ```bash curl 'https:///datasets' @@ -272,7 +273,7 @@ After that, proceed to ⏩[Step 6](#86-validate-your-new-datasets-metadata-recor | :-------------- | :--------------------------------------- | :-------------------------------------- | | `6` | After successfully updating new dataset metadata ⏩[Step 3.2](#832-update-metadata-for-existing-dataset), after adding permissions for the new dataset ⏩[Step 4](#84-add-permissions-for-your-new-dataset) or after making additional declarations for your dataset ⏩[Step 5](#85-make-additional-statements-for-your-dataset). | Node beacon maintainer | -As the node beacon maintainer, proceed to validate your new dataset metadata by running the [beacon verifier](https://beacon-verifier-demo.ega-archive.org/) on your beacon instance. +As the node beacon maintainer, proceed to validate your new dataset metadata by running the [beacon verifier](https://beacon-verifier-demo.ega-archive.org/) on your beacon instance. In case the service is not operative, please, proceed to [download and run the software locally](https://github.com/EGA-archive/beacon-verifier-v2) in order to verify your beacon. Focus on the `/datasets` endpoint. - If the endpoint is valid, record the action in the local audit or change log, including who performed the change, when it was performed, what was changed and the approved scope. - If the endpoint is not valid, record the response obtained from the used commands, adding all the information about the actions performed and the intended goal of performing them and report to the GDI Virtual Helpdesk so that requester communication continues through the VHD workflow. From abffcc0ea499ca25383e90e84b6ad94ff28d6146 Mon Sep 17 00:00:00 2001 From: costero-e Date: Fri, 26 Jun 2026 16:38:21 +0200 Subject: [PATCH 09/10] fixing when for steps 8.5 and 8.6 --- .../GDI-SOP0014_add-datasets-metadata-to-node-beacon.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md b/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md index 54803f3..da89b4d 100644 --- a/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md +++ b/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md @@ -255,7 +255,7 @@ curl 'https:///api/datasets/' | Step identifier | When | Who | | :-------------- | :--------------------------------------- | :-------------------------------------- | -| `5` | After successfully updating new dataset metadata ⏩[Step 3.2](#832-update-metadata-for-existing-dataset), after adding permissions for the new dataset ⏩[Step 4](#84-add-permissions-for-your-new-dataset) or after making additional declarations for your dataset ⏩[Step 5](#85-make-additional-statements-for-your-dataset). | Node beacon maintainer | +| `5` | After successfully updating new dataset metadata ⏩[Step 3.2](#832-update-metadata-for-existing-dataset) or after adding permissions for the new dataset ⏩[Step 4](#84-add-permissions-for-your-new-dataset) | Node beacon maintainer | In case you want your dataset to be declared as meant for test mode, specify its nature or deprecate it, you can by editing the `/beacon/conf/datasets/datasets_conf.yml` file. Add a new entry with the dataset id as the main property and add whatever three optional following items you want to declare for the dataset, setting them as `True`. ```yaml @@ -271,7 +271,7 @@ After that, proceed to ⏩[Step 6](#86-validate-your-new-datasets-metadata-recor | Step identifier | When | Who | | :-------------- | :--------------------------------------- | :-------------------------------------- | -| `6` | After successfully updating new dataset metadata ⏩[Step 3.2](#832-update-metadata-for-existing-dataset), after adding permissions for the new dataset ⏩[Step 4](#84-add-permissions-for-your-new-dataset) or after making additional declarations for your dataset ⏩[Step 5](#85-make-additional-statements-for-your-dataset). | Node beacon maintainer | +| `6` | After adding permissions for the new dataset ⏩[Step 4](#84-add-permissions-for-your-new-dataset) or after making additional declarations for your dataset ⏩[Step 5](#85-make-additional-statements-for-your-dataset). | Node beacon maintainer | As the node beacon maintainer, proceed to validate your new dataset metadata by running the [beacon verifier](https://beacon-verifier-demo.ega-archive.org/) on your beacon instance. In case the service is not operative, please, proceed to [download and run the software locally](https://github.com/EGA-archive/beacon-verifier-v2) in order to verify your beacon. Focus on the `/datasets` endpoint. From cbf7aa18d6ed819ea87de73785535a1298013e0d Mon Sep 17 00:00:00 2001 From: costero-e Date: Fri, 26 Jun 2026 16:44:42 +0200 Subject: [PATCH 10/10] rephrasing step 8.1 --- .../GDI-SOP0014_add-datasets-metadata-to-node-beacon.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md b/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md index da89b4d..45cb074 100644 --- a/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md +++ b/sops/node-specific/GDI-SOP0014_add-datasets-metadata-to-node-beacon.md @@ -122,7 +122,7 @@ After collecting all the information related to the dataset’s metadata, create id name ``` -Fill in the metadata in the row after the header. Ensure that the id follows the required [Dataset SHACL](https://raw.githubusercontent.com/GenomicDataInfrastructure/gdi-metadata/refs/heads/main/Formulasation(shacl)/core/PiecesShape/Dataset.ttl), which needs to be consistent with the SHACL that the Node is currently using in their FDP and doesn't need to be the latest version from the gdi metadata repository. +Fill in the metadata in the row after the header. Ensure that the id follows the required [Dataset SHACL](https://raw.githubusercontent.com/GenomicDataInfrastructure/gdi-metadata/refs/heads/main/Formulasation(shacl)/core/PiecesShape/Dataset.ttl). It should be consistent with the SHACL version currently used by the Node's FDP, which does not necessarily have to be the latest version available in the GDI Metadata Repository. Next step is to tell the ri-tools tool where the metadata file is stored. In `/ri-tools/conf/conf.py` update the value of the `csv_folder` configuration variable so that it points to the folder containing the dataset metadata file, datasets.csv. Example: ```python csv_folder = './csv/'