SNOW-3923354 decouple pandas version from python connector and support pandas 3.x - #4312
Open
sfc-gh-jzeng wants to merge 20 commits into
Open
SNOW-3923354 decouple pandas version from python connector and support pandas 3.x#4312sfc-gh-jzeng wants to merge 20 commits into
sfc-gh-jzeng wants to merge 20 commits into
Conversation
sfc-gh-jzeng
requested review from
sfc-gh-aling,
sfc-gh-bkogan and
sfc-gh-mayliu
August 11, 2026 18:35
|
I have read the CLA Document and I hereby sign the CLA You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4312 +/- ##
==========================================
- Coverage 95.47% 94.88% -0.59%
==========================================
Files 171 171
Lines 44749 44773 +24
Branches 7682 7687 +5
==========================================
- Hits 42723 42484 -239
- Misses 1253 1432 +179
- Partials 773 857 +84 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
![snowflake-security-bot[bot]](https://avatars.githubusercontent.com/u/140554735?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
sfc-gh-jzeng
force-pushed
the
jzeng/snow-3923354-pandas3x-support
branch
from
August 18, 2026 01:52
8caf817 to
166930f
Compare
…as 3 ColumnEmulator replaced pandas.Series._metadata (['_name']) instead of extending it. NDFrame.__finalize__ propagates only the intersection of the two _metadata lists, so every ColumnEmulator copy silently dropped _name and .name became None. The defect is pre-existing and version-independent -- it reproduces on pandas 2.3.1 in isolation. What changed is that pandas 3.0.5 added "grouper = grouper.copy(deep=False)" to Grouping.__init__, which forces that copy on every group-by, so the pivot and group-by family started failing. Upstream: pandas-dev/pandas#61491. Beyond pivot, a nulled .name silently corrupts mock_count_distinct (_functions.py:444 keys a TableEmulator on cols[i].name, collapsing every column onto a single None key with no exception) and _functions.py:962. No test covers either, which is why this is a product bug and not a pivot quirk. TableEmulator._metadata is deliberately left alone: pandas.DataFrame._metadata is empty, so it omits nothing, and adding "_name" there makes the DataFrame->Series __finalize__ intersection non-empty and clobbers correct column labels. tests/integ/scala --local_testing_mode on pandas 3.0.5: 22 failed/595 passed -> 9 failed/608 passed, zero regressions. tests/mock unchanged at 2/462. pandas 2.3.1 stays fully green at 617 passed.
DataFrame.sort_values re-wraps each sort column as pandas.Series(ndarray) before handing it to key= (pandas/core/frame.py; the line is identical in pandas 2 and 3). What changed is Series.__init__ inference: pandas 3 infers the dedicated str dtype for an object array of strings, and str's NA sentinel is nan. So a SQL NULL that is a real None in the frame reached custom_comparator as nan, `value_a is None` stopped firing, and the comparator fell through to a mixed float/str comparison.
pandas 3 infers the dedicated str dtype for an object array of strings and None, and str's NA sentinel is nan—so seven mock-layer sites that deliberately produced None had it silently converted on the way out of apply, combine, DataFrame.T.apply, iterrows, and a bare ColumnEmulator built from a list. Rebuild each result at explicit object dtype; gate the variant sink on isna_helper, since iterrows re-infers per row and has no upstream dtype to preserve.
A chained inplace replace mutates a temporary, which pandas 2 still wrote through to the parent while warning. pandas 3 is Copy-on-Write only, so the write is discarded and only announced via ChainedAssignmentError—a Warning subclass, so nothing raises. MERGE-INSERT columns omitted from the insert clause kept nan instead of None. Assign the result back.
…module__ pandas 3 re-homed read_sql from pandas.io.sql to the top-level pandas namespace, and code_generation routes "from X import Y" by Y.module, so the generated source legitimately changed. Product code is correct; only the expectation was stale. Interpolate the module rather than hard-coding the pandas-3 spelling, which would move the failure to the py310 job that still resolves pandas 2.
pandas 3 removed the integer-as-position fallback on Series.getitem, so iloc[0][0], dtypes[0] and row[0] from iterrows became label lookups that raise KeyError. Test-only: the vectorized UDFs run fine server-side, and product code's row[0] indexes a Snowpark Row, which is a tuple subclass. The accessor fix alone is not sufficient -- it unmasks two expectations that were only hidden because the KeyError fired first. VARCHAR-transported types (STRING, ARRAY, GEOGRAPHY, GEOMETRY, MAP) now arrive as str dtype rather than object, and pandas.Timestamp.module moved to the top-level namespace. Both accept either value, matching the spelling test_pandas_udf_input_types already uses, so the py310 job on pandas 2 keeps passing.
Pandas 3 renames two dtypes this test pins by string: uncast VARCHAR comes back as str instead of object, and the local-testing timestamp default resolution moved from ns to us. Labels only; values and instants are unchanged, so accept either spelling rather than branching on version. The live path already gets its expected timestamp dtype from pyarrow, so only local testing was hit. Verified on pandas 2.3.1 and 3.0.5, offline and against a live account.
write_pandas serializes to parquet, where a timedelta64 column becomes a duration Snowflake does not read back as a duration. The raw tick count lands in a NUMBER, so the unit is baked into the stored value. Pandas 2 inferred ns; pandas 3 infers us. The same timedelta(days=1) therefore started storing 86400000000 instead of 86400000000000 into the same column, with no warning. Nanoseconds is the contract the local testing emulator already enforces via Timedelta.value, and what every table written by an older client holds. The live path only matched that by accident, because ns was pandas' default. Normalizing at the single connector call site covers both write_pandas and create_dataframe(pdf). The to_pandas expectations go the other direction. The live path pins TIMESTAMP_NTZ to datetime64[ns] via pyarrow, while the emulator returns pandas' own default, so they cannot share a literal; the expectation is derived from local_testing_mode. String columns drop dtype=object, since plain inference now matches on both versions and both paths. Only the all-NULL column still needs an explicit dtype, because pd.Series([None]) still infers object on pandas 3.
MAP(STRING, INT) arrives as Arrow map<string, decimal128(38,0)>, so to_pandas() yields an object column of Decimal map values. Pandas 2 serializes that Decimal as a JSON number; pandas 3 emits a JSON string. Only one token changes: 1.0 becomes "1". Despite the test name, this is not a dtype change. The column is object on both versions, verified directly, so the dtype half of the expectation is left alone, as is the non-structured branch, whose values travel as VARIANT text and are byte-identical across versions. We never call DataFrame.to_json in product code; this only affects users who do.
Add `pandas_major_version` to `_internal/utils.py`
Split the unpinned local-testing job so both majors gate the PR.
write_arrow skipped the pandas write path, so a us duration was stored 1000x too small. Also stop calling the local-testing NULL fix pandas-3-only in the changelog.
These three still used Series.combine after the to_char fix, so a string NULL became nan and IS NULL missed the row.
Name the remaining semi-structured and geo string types, and stop saying the str/nan conversion cannot be disabled.
The helper-only Arrow test stays green if write_arrow skips normalize. Pin both write paths and the mock initcap(delimiters) branch.
The timedelta fix applies to any non-nanosecond column, so scoping the entry to pandas 3 told the affected pandas 2 users they were unaffected, and omitted that rows written earlier need correcting. The pandas cap for modin is declared by our own [modin] extra, not by modin itself.
The stored procedure suite uploads our test files to the server but runs them against the snowpark bundled in the Python UDF sandbox, which ships the released version rather than the branch build. A test module that imports a symbol added on this branch therefore cannot be collected at all, losing the whole file instead of skipping one assertion: ImportError: cannot import name 'pandas_major_version' from 'snowflake.snowpark._internal.utils' Jenkins trigger #103 caught this in PythonStoredProcBuildSnowfortTest. GitHub CI cannot, because there the installed snowpark is the branch itself and the import resolves. test_df_to_pandas.py carried the same import and was a latent second instance, masked by SNOW-3674599 skipping it for lack of pandas on Python 3.14. The constant in _internal/utils.py stays for product code, which always runs against its own tree.
sfc-gh-jzeng
force-pushed
the
jzeng/snow-3923354-pandas3x-support
branch
from
August 23, 2026 03:17
166930f to
cf9a8eb
Compare
sfc-gh-jzeng
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which Jira issue is this PR addressing? Make sure that there is an accompanying issue to your PR.
Fixes SNOW-3923354
Fill out the following pre-review checklist:
Please describe how your code solves the related issue.
The connector
[pandas]extra still pinspandas<3. This PR makes Snowpark's[pandas]extra declarepandas<4andpyarrowitself, so a client can install pandas 3. pandas 2 stays supported. A Snowpark bump does not upgrade an existing pandas 2 install.[modin]stays on pandas 2 (pandas<=2.4).Customer BCR on pandas 3:
to_pandas()/to_pandas_batches()use thestrdtype for text columns (VARCHAR,VARIANT,OBJECT,ARRAY,MAP, geo). SQL NULL in those columns comes back asnan, sovalue is Nonemisses. Usepandas.isna.collect()is unchanged.future.infer_string = Falsedoes not turn this off.write_pandas/write_arrow/create_dataframeconvert duration columns to ns before write. Snowflake stores a unit-less integer; the contract is ns. pandas 3 defaults tous.Local testing rebuilds columns as Python lists (
dtype=object) so SQL NULL staysNoneunder pandas 3str+ Copy-on-Write. Precommit local-testing has two jobs: pandas 2 and pandas 3.