diff --git a/.jules/bolt.md b/.jules/bolt.md index d87bd6d..8c3a5bd 100644 --- a/.jules/bolt.md +++ b/.jules/bolt.md @@ -8,3 +8,6 @@ ## 2024-03-29 - ASE Custom JSON encoding vs standard JSON **Learning:** ASE's custom JSON encoder (`ase.io.jsonio.encode`) will generate dicts with special keys like `__ndarray__` or `__complex__` (e.g. `{"__ndarray__": [[5], "int64", ...]}`). When optimizing JSON deserialization using faster alternatives like `orjson`, it's critical to realize that a normal `json.loads` or `orjson.loads` will deserialize this into a Python dictionary, while ASE's custom `decode` will properly reconstruct the underlying numpy array. Bypassing ASE's decoder without checking for these keys leads to downstream type errors (e.g. `KeyError: '__ndarray__'`). **Action:** When replacing or wrapping ASE's jsonio with `orjson`, always fall back to ASE's `decode` if the payload string contains `__ndarray__` or `__complex__` markers, to ensure custom objects are correctly reconstructed. +## 2024-05-19 - Replacing iterrows with to_dict('records') +**Learning:** In data handling files like `verify_processed_omol25.py`, iterating over large Pandas DataFrames using `df.iterrows()` is an anti-pattern that creates significant bottlenecks because it instantiates a Pandas Series object for every single row. A highly performant alternative is to convert the DataFrame to a list of dicts using `df.to_dict('records')` first. +**Action:** When iterating over a pandas DataFrame, especially large ones, use `df.to_dict('records')` instead of `df.iterrows()`. Remember to update downstream method calls on the row object (e.g., replacing `row.to_dict()` with `dict(row)` or just `row`), as the row is now a native Python dictionary instead of a Pandas Series. diff --git a/src/lavello_mlips/verify_processed_omol25.py b/src/lavello_mlips/verify_processed_omol25.py index 9d1c47c..797b4e7 100644 --- a/src/lavello_mlips/verify_processed_omol25.py +++ b/src/lavello_mlips/verify_processed_omol25.py @@ -77,8 +77,12 @@ def main() -> None: ) logger.info(f"Loaded {len(df)} records from Parquet.") - parquet_by_sha = {row["geom_sha1"]: row for _, row in df.iterrows()} - parquet_by_argone_rel = {row["argonne_rel"]: row for _, row in df.iterrows()} + # Performance Optimization: Convert DataFrame to a list of dicts. + # Iterating over native Python dicts is ~10x faster than using df.iterrows() + # because it avoids the overhead of instantiating a Pandas Series for every row. + records = df.to_dict("records") + parquet_by_sha = {row["geom_sha1"]: row for row in records} + parquet_by_argone_rel = {row["argonne_rel"]: row for row in records} logger.info(f"Loading ExtXYZ file from {args.extxyz} (this may take a moment)...") all_atoms = read(str(args.extxyz), index=":") if not isinstance(all_atoms, list): @@ -108,7 +112,9 @@ def get_dump_entry(at): info = dict(at.info) rel = info.get("argonne_rel") pq_row = parquet_by_argone_rel.get(rel) - pq_data = pq_row.to_dict() if pq_row is not None else None + # Performance Optimization: pq_row is now a native dict instead of a Pandas Series, + # so we use dict(pq_row) instead of pq_row.to_dict(). + pq_data = dict(pq_row) if pq_row is not None else None return {"xyz": info, "parquet": pq_data} duplicates = {