Skip to content

Fix upserting when Synapse column name contains space or special characters - #1454

Merged
andrewelamb merged 1 commit into
developfrom
upsert_fixes
Sep 18, 2026
Merged

andrewelamb merged 1 commit into
developfrom
upsert_fixes

Conversation

@andrewelamb

@andrewelamb andrewelamb commented Sep 17, 2026 •

Copy link
Copy Markdown
Contributor

Problem:

upsert_rows misreports how many rows need to be updated when a Synapse
table or view has a column name that is not a valid Python identifier, such
as a name with a space (AMP AIM_SSc-Skin_Access Team) or a hyphen
(col-with-hyphen).

  • Reported by Jessica Vera during QA testing of SYNPY-1912
    (against syn76143756, a table with several columns whose names contain
    spaces).
  • _construct_partial_rows_for_upsert in
    synapseclient/models/mixins/table_components.py loops over the freshly
    queried rows with results.itertuples(index=False). itertuples builds a
    namedtuple from the column names, but a namedtuple field name must be a
    valid Python identifier. When a column name has a space or another
    special character, pandas silently swaps in a fallback name for that
    field (e.g. _5), so the real column name is no longer reachable via
    getattr/hasattr.
  • The code used getattr(row, column) if hasattr(row, column) else None to
    read each cell, and if not hasattr(row, column): values_differ = True
    to decide whether a row changed. Since hasattr was always False for
    any column with a space or special character, every row was treated as
    changed for that column, on every call, even when nothing changed.
  • This made every upsert against such a table report the full row count as
    "rows to update," regardless of how many rows actually changed.
  • This bug pre-dates SYNPY-1912 (introduced in [GEN-2381] Pandas handling of nullable cells #1272, 2025-11-20). SYNPY-1912
    ([SYNPY-1912] Fix upsert_rows misreporting Table update responses #1446) fixed a separate, downstream issue in how update results were
    logged, but did not touch this code path.

Solution:

  • Replaced attribute-style row access (getattr/hasattr on the
    itertuples result) with position-based access, which works regardless
    of what pandas named the namedtuple field.
  • Built a column_positions map ({column_name: index}) from
    results.columns once per call, and used it to look up ROW_ETAG,
    ROW_ID, id, the primary key columns, and each compared column by
    position (row[column_positions[...]]) instead of by name.
  • This is a minimal, targeted fix: it keeps the existing itertuples loop
    and comparison logic, and only changes how values are read out of each
    row.

Testing:

  • Added
    test_construct_partial_rows_for_upsert_with_column_name_containing_special_characters
    to tests/unit/synapseclient/mixins/unit_test_table_components.py. It
    exercises columns named col with space and col-with-hyphen with
    identical values in the queried results and the upsert data, and asserts
    that no rows are reported as changed. This test failed before the fix
    (assert 2 == 0) and passes after it.
  • Ran the full existing test file
    (tests/unit/synapseclient/mixins/unit_test_table_components.py): all 336
    tests pass, including the 13 pre-existing tests that call
    _construct_partial_rows_for_upsert directly, so no regressions.
  • Verified against the real table (syn76143756) that its column names do
    in fact contain spaces and special characters (e.g.
    AMP AIM_SSc-Skin_Access Team), confirming this fix addresses the
    reported case.

@andrewelamb
andrewelamb requested a review from a team as a code owner September 17, 2026 17:51
@andrewelamb
andrewelamb marked this pull request as draft September 17, 2026 17:52
@andrewelamb andrewelamb changed the title fix upserting when Synapse column name contains space or special char Fix upserting when Synapse column name contains space or special char Sep 17, 2026
@andrewelamb andrewelamb changed the title Fix upserting when Synapse column name contains space or special char Fix upserting when Synapse column name contains space or special characters Sep 17, 2026
@andrewelamb
andrewelamb marked this pull request as ready for review September 17, 2026 18:05
@andrewelamb
andrewelamb merged commit 74cc58f into develop Sep 18, 2026
29 of 30 checks passed
@andrewelamb
andrewelamb deleted the upsert_fixes branch September 18, 2026 18:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants