New entity type: General data set

[MWolfR]

Description

A new entity is generated for general data sets that do not fit into any of the existing categories.
Support should exist in entitysdk for the new entity.

Technical implementation & specs

• Child of ScientificArtifact
• Asset is a single directory asset where the user can put whatever they want with little restriction (total size should be restrictred though)
• Maybe another .json asset that specifies the structure of the directory asset and the sizes of the contained files. Unless this info is easily queried through entitycore.
• Additional properties:
• data_category (required): One of "Experimental", "Model", "Simulation". Determines where it will be displayed.
• documentation_url (optional): A link to documentation about the data or its format. E.g. link to a repository for interacting with and loading the data
• documentation_path (optional): A path within the directory asset that points to a readme text or .md file.

Describe alternatives considered

Maybe instead of a directory asset it could be a single .zip file asset (the zip file in turn could contain a directory structure...)

Acceptance criteria

• Schema designed and agreed on
• Implemented in entitycore and deployed
• Implemented in entitysdk

[MWolfR]

Exemplary use case:

1. In a first notebook, we load the MICrONS circuitry and analyze it, calculating some measurement for all neurons. Based on that measurement, 15 neurons are selected.
2. For the selected neurons, I run the Skeletonization workflow
3. In a second notebook, I load the skeletonized morphologies and analyze them. At this point I need the results of the measurement from step (1.) for context.

Currently, I simply write the measurement results and the ids of the selected neurons into a file on JupyterHub. That works, but for a demo it seems weak: The measurements taken are a result in their own right and should be treated as such.

[MWolfR]

Another use case:
Store the mean, standard deviation and probability distribution type for synapse parameters. These then serve as an input for a prototype workflow that adds the parameters to an anatomical circuit model. This is for prototyping, until a better, specific entity for the purpose is defined and implemented.

[MWolfR]

After discussion we came to the conclusion that this is different in scope and purpose from #579
So implementation of this can move forward.

[jdcourcol]

I have additional questions (cc: @james-isbister & @GianlucaFicarelli ):

• Do we plan to use that as input in workflows ? I am a bit afraid of several workflow using a "general" dataset as an input. Then you don't really know what is usable or not.
• Do we plan to have the same lifecycle as regular entities ( any special requirement for deletion) ?
• Do we plan to have the same relationship such as derivation, usage ?

[james-isbister]

So we discussed two different types of generic data/results entities (both of which we think would be useful for different scenarios):

1. A very general entity with a single directory asset (as described by @MWolfR for this ticket). This could be for holding any output of a notebook, or for uploading data to the platform (say as the input to a circuit registration workflow or just to initially store a large data upload, say from LNMC, before we process it into dedicated entities).
2. An entity which follows the pattern we have TaskConfig and TaskActivity, with a type enum (i.e. simulation_result, spike_sorting_result, etc), which would each allow specific asset keys (i.e. for simulation_result: spike_report, voltage_report etc).

The second (stricter) entity type would be more likely to be used as the input to specific workflows, although there is the case of an entity registration workflow for the first.

I guess the second type would be registered as a "generated" entity of the activity. I'm not sure about its lifecycle though. The goal of the second type is to allow very quick introduction of new workflows. For example, for the Meta Tribe V2 MRI foundation model use case, the second type would allow us to very quickly define asset keys for the output of the workflow and register the results.

There is an interesting difference though that it would probably be for cases where the workflow creates a "result" or "analysis result", rather than a "tangible" new entity like an electrode weights array or circuit

This discussion might be better suited to the capability ticket #579

[jdcourcol]

Do we plan to use that as input in workflows ? I am a bit afraid of several workflow using a "general" dataset as an input. Then you don't really know what is usable or not.
Do we plan to have the same lifecycle as regular entities ( any special requirement for deletion) ?
Do we plan to have the same relationship such as derivation, usage ?

I still have these questions, @james-isbister @MWolfR