Develop merge - #58
Merged
Merged
Conversation
☂️ Python Coverage
Overall Coverage
New Files
Modified FilesNo covered modified files...
|
…ds to be tested/debugged
[WIP] Feature/genome vectorize
Mp/netvae refactor
[WIP] NetVAE refactor
Contributor
Author
|
Limiting the scope of this PR, to keep it from staying here forever |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR is a broad refactor/feature expansion that adds pathway-constrained model tooling (constraints, mask enforcement, verification, CLI workflows), expands test coverage (unit + e2e), and introduces initial genomic vectorization utilities (VCF bin-count vectorization), alongside logging/UX improvements across CLI and utilities.
Changes:
- Adds a new pathway constraint system (
PathwayConstraintInfo) and strengthens masked-weight enforcement (post-step + pre-save clamping) with model integrity verification/CLI support. - Refactors NetVAE construction/serialization and updates pathway parsing/mask-building utilities and related tests/e2e workflows.
- Introduces genome variant-count vectorization scaffolding and adds multiple new tests/resources/docs and CI coverage workflow updates.
Reviewed changes
Copilot reviewed 74 out of 76 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/resources/test_resource.py | Tests EMBKIT_HOME default path behavior |
| tests/resources/test_resource_base.py | Tests default save-path creation via env |
| tests/resources/test_resource_and_gtex.py | Tests GTEx + default path via env |
| tests/resources/test_c_bio_portal.py | Stabilizes default path tests (no HOME writes) |
| tests/pathway/test_pathway_mask.py | Updates mask tests for new pathway APIs |
| tests/pathway/test_layer_helpers.py | Repoints helper tests to embkit.pathway |
| tests/optimize/test_optimize_helpers.py | Adds coverage for optimize helpers |
| tests/modules/test_tsp.py | Adds tests for TSPLayer behavior |
| tests/models/vae_models/test_rna_vae.py | Adds RNAVAE smoke-fit test |
| tests/models/vae_models/test_net_vae.py | Updates NetVAE tests; adds constraint-phase test |
| tests/models/vae_models/test_encoder.py | Migrates constraint test to PathwayConstraintInfo |
| tests/files/test_loaders.py | Adds tests for file loaders |
| tests/files/test_large_csv_reader.py | Adds tests for LargeCsvReader |
| tests/files/test_h5.py | Extends H5 tests incl. cube readers |
| tests/factory/test_verify.py | Adds tests for model verification reports |
| tests/factory/test_vae.py | Adds NetVAE roundtrip + constraint dispatch tests |
| tests/factory/test_layers_helpers.py | Adjusts layer helper test scaffolding |
| tests/encoding/test_protein_encoder.py | Adds tests for ProteinEncoder wrapper |
| tests/encoding/test_mutation.py | Adds tests for variant-count vectorization |
| tests/encoding/init.py | Ensures unittest discovery in encoding tests |
| tests/e2e/test_netvae_cli_e2e.py | Adds CLI e2e for train-netvae + encode |
| tests/e2e/test_cli_workflows_e2e.py | Adds CLI e2e for multiple workflows |
| tests/e2e/data/toy_pathway.sif | Adds toy pathway fixture |
| tests/e2e/data/toy_expr.tsv | Adds toy expression fixture |
| tests/e2e/init.py | Ensures e2e unittest discovery |
| tests/datasets/test_datasets.py | Adds dataset utility tests |
| tests/datasets/init.py | Ensures unittest discovery in datasets |
| tests/constraints/test_pathway_constraints.py | Adds tests for PathwayConstraintInfo |
| tests/constraints/test_network_constraints.py | Removes legacy NetworkConstraint tests |
| tests/commands/test_resources.py | Adds CLI tests for resource commands |
| tests/commands/test_protein.py | Adds CLI tests for protein commands |
| tests/commands/test_model.py | Adds CLI tests for model commands |
| tests/commands/test_matrix.py | Adds CLI tests for matrix commands |
| tests/c_bio/test_c_bio_api.py | Switches CBIO API error reporting to logging |
| src/embkit/utilities/pca.py | Uses logging; fixes CSV header handling |
| src/embkit/resources/resource.py | Adds EMBKIT_HOME override + improved error logging |
| src/embkit/preprocessing/normalize.py | Adds typing and return annotations |
| src/embkit/pathway.py | Refactors SIF parsing + mask/feature-map utilities |
| src/embkit/optimize/init.py | Adds mask clamping; phases/logging tweaks |
| src/embkit/modules/tsp.py | Replaces prints with logging; marks main as no-cover |
| src/embkit/modules/masked_linear.py | Adds clamping API + in/out feature properties |
| src/embkit/models/vae/vae.py | Persists history in serialization |
| src/embkit/models/vae/rna_vae.py | Adds verify_integrity + serialization; logging cleanup |
| src/embkit/models/vae/net_vae.py | Rebuilds NetVAE around pathway constraints + audits |
| src/embkit/models/vae/encoder.py | Switches to generic ConstraintInfo masks |
| src/embkit/models/vae/decoder.py | Improves decoder build bookkeeping/logging |
| src/embkit/models/vae/base_vae.py | Adds generic model integrity verification logic |
| src/embkit/models/ffnn.py | Adds serialization + integrity verification |
| src/embkit/losses/vae_loss.py | Chooses BCE vs BCE-with-logits based on output range |
| src/embkit/files/read_csv.py | Replaces prints with logging; CI tqdm disable; read fixes |
| src/embkit/files/h5.py | Refactors H5 readers; adds cube writer/reader utilities |
| src/embkit/factory/layers.py | Introduces ConstraintInfo ABC + masked layer build changes |
| src/embkit/factory/core.py | Adds save-time clamping + verification runner |
| src/embkit/factory/init.py | Exports run_model_verification |
| src/embkit/encoding/genome.py | Adds genome variant count vectorization utility |
| src/embkit/encoding/init.py | Adds encoder shape metadata + serialization hooks |
| src/embkit/constraints/pathway_constraint.py | Adds PathwayConstraintInfo implementation |
| src/embkit/constraints/network_constraint.py | Removes legacy NetworkConstraint implementation |
| src/embkit/constraints/init.py | Updates constraint exports to PathwayConstraintInfo |
| src/embkit/commands/protein.py | Improves file handling; avoids closing stdout |
| src/embkit/commands/model.py | Updates train-netvae flow; adds model verify |
| src/embkit/commands/matrix.py | Fixes feature subset handling; adds matrix pca |
| src/embkit/commands/align.py | Updates output formatting for new align return type |
| src/embkit/c_bio/api.py | Uses logger instead of print for errors |
| mkdocs.yml | Updates nav; removes NetworkConstraint docs entry |
| docs/training.md | Documents history persistence + constraint-safe save |
| docs/requirements.txt | Adds mkdocs requirement |
| docs/examples/netvae.md | Updates NetVAE example + adds verify usage |
| docs/concepts.md | Updates constraint docs + explains clamping rationale |
| docs/cli.md | Documents model verify + matrix pca + NetVAE options |
| docs/change-notes.md | Adds change notes summary page |
| docs/api/models/net_vae.md | Updates API docs member list |
| docs/api/factory/index.md | Documents verification + save-time clamping |
| docs/api/constraints/network_constraint.md | Removes legacy constraint doc page |
| docs/api/constraints/index.md | Updates to PathwayConstraintInfo canonical doc |
| .github/workflows/pr_coverage_check.yml | Overhauls PR coverage reporting/enforcement |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+156
to
+158
| @click.option("--min-group-size", type=int, default=0, show_default=True, help="Minimum group size filter for pathway feature map (including self if present).") | ||
| @click.option("--group-layer-scale", default="5,2,1", show_default=True, | ||
| help="Comma-separated per-group widths for NetVAE masked layers.") |
Comment on lines
+171
to
+175
| feature_map = extract_sif_interactions(pathway_sif) | ||
| feature_map = feature_map_intersect(feature_map, df.columns, min_group_size=min_group_size) | ||
| feature_idx, group_idx = build_feature_map_indices(feature_map) | ||
|
|
||
| feature_map, isect = feature_map_intersect(feature_map, df.columns) | ||
|
|
||
| df = df[isect] | ||
| df = df[feature_idx] |
Comment on lines
+35
to
+52
| def __init__( | ||
| self, | ||
| features: List[str], | ||
| latent_groups: Dict[str, List[str]], | ||
| latent_index: Optional[List[str]] = None, | ||
| group_layer_scale: Optional[List[int]] = None, | ||
| batch_norm: bool = False, | ||
| device: Optional[torch.device] = None, | ||
| dtype: Optional[torch.dtype] = None, | ||
| ): | ||
| if not latent_groups: | ||
| raise ValueError("latent_groups cannot be empty for NetVAE.") | ||
|
|
||
| if group_layer_scale is None: | ||
| group_layer_scale= [1, 1] | ||
| group_layer_scale = [int(v) for v in group_layer_scale] | ||
| if any(v <= 0 for v in group_layer_scale): | ||
| raise ValueError(f"group_layer_scale must contain positive integers, got {group_layer_scale}.") |
Contributor
Author
There was a problem hiding this comment.
@copilot, rather then change the API, can you update the docs to match the code?
Comment on lines
+266
to
292
| def to_dict(self) -> Dict[str, Any]: | ||
| return { | ||
| "features": self.features, | ||
| "encoder": self.encoder.to_dict() if self.encoder else None, | ||
| "decoder": self.decoder.to_dict() if self.decoder else None, | ||
| "latent_index": self.latent_index, | ||
| "latent_groups": self.latent_groups, | ||
| "latent_index": self.latent_index, | ||
| "group_layer_scale": self.group_layer_scale, | ||
| "history": getattr(self, "history", {}) or {} | ||
| } | ||
|
|
||
| @classmethod | ||
| def from_dict(cls, d): | ||
| features = d.get("features") | ||
| if features is None: | ||
| fmap = d.get("latent_groups") or {} | ||
| feature_set = set() | ||
| for members in fmap.values(): | ||
| feature_set.update(members) | ||
| features = sorted(feature_set) | ||
|
|
||
| model = NetVAE( | ||
| features=d["features"], | ||
| encoder=Encoder.from_dict(d["encoder"]) if d.get("encoder") else None, | ||
| decoder=Decoder.from_dict(d["decoder"]) if d.get("decoder") else None, | ||
| features=features, | ||
| latent_groups=d.get("latent_groups"), | ||
| latent_index=d.get("latent_index"), | ||
| group_layer_scale=d.get("group_layer_scale"), | ||
| ) | ||
| model.latent_index = d.get("latent_index") | ||
| model.latent_groups = d.get("latent_groups") | ||
| model.history = d.get("history") or {} | ||
| return model |
Comment on lines
+157
to
+168
| def build_feature_map_indices( | ||
| feature_map: Dict[str, List[str]]) -> Tuple[pd.Index, pd.Index]: | ||
| """ | ||
| Create feature index from a feature map | ||
| """ | ||
| feature_set = set() | ||
| group_set = sorted(feature_map.keys()) | ||
| for group in group_set: | ||
| feature_set.update(feature_map[group]) | ||
| feature_idx = pd.Index( sorted(feature_set) ) | ||
| group_idx = pd.Index(group_set) | ||
| return feature_idx, group_idx |
Comment on lines
+57
to
+67
| bins = [] # (chrom, bin_start, bin_end, bin_label) | ||
| bin_labels=[] | ||
|
|
||
| counter=0 | ||
| for chrom, length in chromosome_length.items(): | ||
| for i in range(1, length, bin_size): # last bin for each chromosome might not equal 1MB depending on the chr length | ||
| bin_label = f'{chrom}_{counter:04d}' | ||
| bins.append((chrom, i, min(i + bin_size, length), bin_label)) | ||
| bin_labels.append(bin_label) | ||
| counter+=1 | ||
| counter = 0 |
Comment on lines
+13
to
+18
| if className not in CLASS_REGISTRY: | ||
| try: | ||
| from .base_vae import _import_obj | ||
| _import_obj(className) | ||
| except Exception as e: | ||
| pass |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tools for creating vectors from genomic data