Optimize radar and lidar forward operators - #163
Merged
Conversation
…idar_micro Introduces ProcessPoolExecutor as a third parallelism mode alongside the existing Dask (parallel=True) and serial (parallel=False) paths. Primary Changes: - Added ProcessPoolExecutor + os + tqdm imports to radar_moments.py and lidar_moments.py - Updated parallel parameter to accept bool | str; 'processes' routes to ProcessPoolExecutor, True/'dask' retains existing Dask bag logic - Applied to all 4 parallel blocks in calc_radar_micro (Ze/Vd pass + spectral width pass) and the single parallel block in calc_lidar_micro - Replaced all 5 local lambdas with module-level _PoolWorker callable class: standard pickle cannot serialize closures, which caused AttributeError when parallel='processes'; _PoolWorker also eliminates implicit capture of model/ instrument objects in cloudpickle, yielding significant speedup under parallel=True (Dask) No algorithm changes; backward compatible (True still uses Dask, False still serial).
… computation Cache N_0, lambdas, mu parameters from the main reflectivity/ velocity pass (pass 1) and reuse them in the spectral width pass (pass 2), eliminating the second call to calc_and_set_psd_params per hydrometeor class.
Add single-pass spectral width computation using variance decomposition formula, eliminating the expensive second parallel pass when calc_spectral_width=True. Primary changes: - Modify worker functions (_calculate_observables_liquid, _calculate_other_observables) to compute v2_numer (velocity-squared integrals) during pass 1 - Accumulate v2_numer across hydrometeor types in calc_radar_micro - Compute total spectral width from variance formula: σ_d² = E[v²] - (E[v])² after Vd_tot is finalized. - Expose parameter in make_simulated_data (main.py) with default=True - Backward compatible: disable with single_pass_spectral_width=False
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces significant improvements to the parallelism options and radar spectral width calculation in the lidar and radar simulators. This is primarily the result of a more efficient single-pass computation for radar spectral width, and several code cleanups and refactorings.
These changes make the simulator more flexible, efficient, and user-friendly for both small and large-scale simulations. From preliminary tests, these changes cut sigma_D calcuation by an order of magnitude, with an estimated total speedup by a factor of 4.