Add reproducible GWAS fixtures and relational manifests

[lyh970817]

Adds a reproducible Nextflow workflow for generating compact GWAS test data across all 22 autosomes. The generated fixtures provide PLINK 1, PLINK 2 and indexed VCF representations together with quantitative and binary phenotypes, covariates and generation provenance.

Also adds relational cohort and analysis manifests, heterogeneous method options, and their small GCTA and LDAK resource files for use by nf-core/gwas.

[chriswyatt1]

Hi.

So this is great. I can see that we don't have anything similar enough for GWAS already on test datasets. Although I think we can reduce the total size down quite a lot, as some files don't seem to be necessary, but I may be missing that point!

[lyh970817]

Hi.

So this is great. I can see that we don't have anything similar enough for GWAS already on test datasets. Although I think we can reduce the total size down quite a lot, as some files don't seem to be necessary, but I may be missing that point!

Yes I can also see some potential in cutting them down. I could use in-flight derivation for the bed/pgen format, and we maybe not need the 22 chromosomes (2 is theoretically enough). And if I update the multi-chromosome plink test files in the modules branch (right now my methods don't converge on them) we might not even need these chromosome files here at all.

I left them all there for completeness essentially - thought it might be more intuitive for future developers to see them all here. I'm wondering how much we should cut down. Happy to do a prune to bare minimum if that's desirable.

[chriswyatt1]

Yes, that makes sense. The test data for the pipeline don't necessarily need to have a logical result. Just the bare minimum to show that the pipeline has completed that task. So if just 1 or 2 chromosomes is enough, go with that. We also want it to be as quick as possible, to reduce all the CI checks for the whole pipeline. So yes, if files can be generated quickly, that is even better, saves the cost of storage.

[chriswyatt1]

Maybe two chromosomes could be nice, I see the files are pretty small, so if you wish to have something bigger, it is fine, if needed. Will need approval from an nf-core core member. Worth checking other repos about size

[lyh970817]

Maybe two chromosomes could be nice, I see the files are pretty small, so if you wish to have something bigger, it is fine, if needed. Will need approval from an nf-core core member. Worth checking other repos about size

A few commits will land shortly to wrap this up. I went with the approach to include only a single vcf and dynamically splitting it into two chromosomes at test launch.

The rationale is that in the future when everything is settled we might probably want to update the generic GWAS-related test datasets in the modules branch to work with the pipeline, so we won't need any actual test datasets here. This should be the cleanest?

A big chunk of the consumers of those generic test datasets are modules written by me, so updating all of the snapshots in them should be easy, but just needs reviews.

[lyh970817]

I'll remove most of the original test datasets as well if that's okay - since if in the future we can use the generic modules test datasets they will also become obsolete. This way that single vcf (plus some converter setup as modules has all formats) will be the only things we will need to retire.

Documentation has not been updated yet - can do that once we've agreed on the test data setup.

[lyh970817]

On the time it takes for the pipeline CI run, on my machine it takes 20~ minutes with 3 shards (so 1 hour if sequential). Each test process requests 1G of memory - I wonder if this is reasonable.

[chriswyatt1]

Hey, great, yes that sounds much better. And 20 minutes seems reasonable to me. I think genomeqc takes ~10-15 minutes on the smallest -- profile test dataset (if I remember correctly), sarek takes more (i think). Good enough to quickly test whe adding new models.

[lyh970817]

Hey, great, yes that sounds much better. And 20 minutes seems reasonable to me. I think genomeqc takes ~10-15 minutes on the smallest -- profile test dataset (if I remember correctly), sarek takes more (i think). Good enough to quickly test whe adding new models.

Then I think it sounds about right - I actually have a dozen of profiles because of the combinations of different methods.

It's good for another review I think - now the PR changed only about 20 files and the rest is to remove the original ones.