Repository navigation
docs: revise and improve QC pages, add new tag widget - #105
Conversation
dougollerenshaw
left a comment
dougollerenshaw
left a comment
There was a problem hiding this comment.
I left a couple inline comments. I think this is an improvement regardless, so I'm approving regardless of whether you address my comments.
If possible, I'd suggesting adding some scientists as reviewers since they're the ones who will actually be interacting with this concept most directly. I want to make sure that they find this sufficiently well documented.
As a side note why are the same changes showing up in both docs/base/core/quality_control.md and docs/source/quality_control.md?
| The metrics defined during quality control should define whether or not an asset can be used for analysis and the properties of an asset that could influence how an analysis is performed. There are two levels of QC: | ||
|
|
||
| Each [QCMetric](#qcmetric) is a single value or array of values that can be computed, or observed, about one modality in a data asset. These can have any type. Metrics should be significant: i.e. whether they pass or fail should matter for the modality. Metrics need to be human understandable. If you find yourself generating more than fifty metrics for a modality you should group them together (i.e. make the value a dictionary combining similar metrics and the rule an evaluation of multiple fields in the dictionary). | ||
| - Quality control metrics that are **not allowed to fail** are metrics that at `stage:raw` would prevent an asset from being processed or that at `stage:processing` would prevent an asset from being analyzed. This is a very high bar. Another way of saying this is that if the goals of the data acquisition were met, then all metrics that are 'not allowed to fail' should be passing. |
There was a problem hiding this comment.
I think very concrete examples of these two types of metrics would be helpful. I'm a little afraid that 'if the goals of the data acquisition were met' is a little bit open to interpretation. You might add something like "For example, if the goal of the acquisition was to collect behavior and fiber photometry, the only 'not allowed to fail' metrics should be metrics that define whether the behavior and fiber photometry data could be analyzed, not whether they pass some thresholds on behavior engagement, signal to noise ratio, or other potentially variable metrics"
|
|
||
| Each [QCMetric](#qcmetric) is a single value or array of values that can be computed, or observed, about one modality in a data asset. These can have any type. Metrics should be significant: i.e. whether they pass or fail should matter for the modality. Metrics need to be human understandable. If you find yourself generating more than fifty metrics for a modality you should group them together (i.e. make the value a dictionary combining similar metrics and the rule an evaluation of multiple fields in the dictionary). | ||
| - Quality control metrics that are **not allowed to fail** are metrics that at `stage:raw` would prevent an asset from being processed or that at `stage:processing` would prevent an asset from being analyzed. This is a very high bar. Another way of saying this is that if the goals of the data acquisition were met, then all metrics that are 'not allowed to fail' should be passing. | ||
| - Quality control metrics that are **allowed to fail** are metrics that will influence the analysis of data, for example by indicating that one element of an asset cannot be used. These assets do not invalidate further analysis but they change how analysis should be performed. |
There was a problem hiding this comment.
This is actually unclear to me. By "element", do you mean something different than "modality"?
Back to the behavior + fiber photometry example. Wouldn't there be different "not allowed to fail" metrics on both modalities independently? In other words, wouldn't it be possible to completely fail (and throw out) the fiber data, but retain the behavior data? I'd assume so, but in that case, I don't know why we'd call that "allowed to fail" so I'm assuming that's different than what you're describing here.
2 participants
This PR improves the QC documentation based on some learning we've done in the past few months about what metrics rise to the level of "critical" QC that should fail an asset.
I also added a cool widget that should help people with developing their QC metric tag hierarchy, which was previously an arcane ask of scientists/engineers.
📚 Documentation preview 📚: https://biodata-schema--105.org.readthedocs.build/en/105/