Decouple heavy plugins from core to optimize for serverless deployment#244
Open
google-labs-jules[bot] wants to merge 1 commit into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context & Rationale
The primary goal of this PR is to transform the
eegprepcore package into a lightweight orchestrator. Currently, mandatory dependencies on machine learning frameworks like PyTorch and CUDA-related libraries result in multi-gigabyte installation sizes. This overhead creates significant friction for serverless deployments, causing slow cold starts and exceeding resource limits for headless scaling.By moving ICLabel and ASR (clean_rawdata) into independent distribution units, we significantly reduce the default footprint while maintaining a modular architecture that allows researchers to opt-in to heavy dependencies as needed.
Key Changes
1. Dependency & Package Refactoring
torchfrom the coreoptional-dependenciesto prevent automatic downloads of ML frameworks during a standard install.src/eegprep/plugins/ICLabelandsrc/eegprep/plugins/clean_rawdata. These are now intended to be hosted in separate repositories.rand_permutationinrunica.py) to the core module to ensure base functionality remains intact without the plugins.2. Plugin Discovery & Runtime Loading
menu_actions.pyandcli/commands/transforms.pyto load plugin-specific logic only when requested, ensuring the headless CLI boot time remains sub-second.3. Improved User Experience
__init__.pywith a custom__getattr__implementation. If a user attempts to call legacy functions likepop_iclabelwithout the extension installed, they receive a descriptiveRuntimeErrorwith specific installation instructions:[all]install extra inpyproject.tomlto support the existing comprehensive installation workflow for local researchers.Success Metrics
eegprepno longer triggers PyTorch or CUDA downloads.Verification Results
test_extensionsandtest_plugin_menusuites passed.