Skip to content

Decouple heavy plugins from core to optimize for serverless deployment#244

Open
google-labs-jules[bot] wants to merge 1 commit into
developfrom
jules/modular-plugin-decoupling-js1-3d9fcb8d-6bb2-43b7-b95b-661f43c00bae
Open

Decouple heavy plugins from core to optimize for serverless deployment#244
google-labs-jules[bot] wants to merge 1 commit into
developfrom
jules/modular-plugin-decoupling-js1-3d9fcb8d-6bb2-43b7-b95b-661f43c00bae

Conversation

@google-labs-jules

Copy link
Copy Markdown

Context & Rationale

The primary goal of this PR is to transform the eegprep core package into a lightweight orchestrator. Currently, mandatory dependencies on machine learning frameworks like PyTorch and CUDA-related libraries result in multi-gigabyte installation sizes. This overhead creates significant friction for serverless deployments, causing slow cold starts and exceeding resource limits for headless scaling.

By moving ICLabel and ASR (clean_rawdata) into independent distribution units, we significantly reduce the default footprint while maintaining a modular architecture that allows researchers to opt-in to heavy dependencies as needed.

Key Changes

1. Dependency & Package Refactoring

  • Removed heavy dependencies: Excised torch from the core optional-dependencies to prevent automatic downloads of ML frameworks during a standard install.
  • Plugin Decoupling: Deleted src/eegprep/plugins/ICLabel and src/eegprep/plugins/clean_rawdata. These are now intended to be hosted in separate repositories.
  • Parity Maintenance: Migrated essential shared utilities (e.g., rand_permutation in runica.py) to the core module to ensure base functionality remains intact without the plugins.

2. Plugin Discovery & Runtime Loading

  • Entry Point Integration: Implemented standardized entry point discovery. The core library now scans the environment at runtime to register externally installed plugins.
  • Lazy Loading: Updated menu_actions.py and cli/commands/transforms.py to load plugin-specific logic only when requested, ensuring the headless CLI boot time remains sub-second.

3. Improved User Experience

  • Graceful Degradation: Updated __init__.py with a custom __getattr__ implementation. If a user attempts to call legacy functions like pop_iclabel without the extension installed, they receive a descriptive RuntimeError with specific installation instructions:
    pip install eegprep-iclabel
  • "All-in-one" Support: Maintained the [all] install extra in pyproject.toml to support the existing comprehensive installation workflow for local researchers.

Success Metrics

  • Footprint Reduction: A clean installation of eegprep no longer triggers PyTorch or CUDA downloads.
  • Performance: Headless agents initialize significantly faster on standard serverless platforms.
  • Compatibility: Verified that the CLI successfully identifies and loads plugins when they are present in the environment via entry points.

Verification Results

  • All test_extensions and test_plugin_menu suites passed.
  • Verified that core scientific operations (MNE/SciPy dependent) function correctly in an environment devoid of the decoupled plugins.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants