Skip to content

Add Decimal256 and Interval(MonthDayNano) support - #332

Open
abdolence wants to merge 4 commits into
chmp:mainfrom
abdolence:decimal256-interval
Open

abdolence wants to merge 4 commits into
chmp:mainfrom
abdolence:decimal256-interval

Conversation

@abdolence

Copy link
Copy Markdown

Summary

Hey, I'm working on a BigQuery client that reads typed rows through the Storage Read API. It sends BIGNUMERIC as Decimal256(76, 38) and INTERVAL as Interval(MonthDayNano), and from_record_batch fails on both today (Unsupported arrow data type Decimal256(76, 38)). The error is for the whole batch, even if the struct doesn't name these columns.

This PR adds:

  • Decimal256, handled the same way as Decimal128. marrow gets the data type, array and view with an i256 element type in the Arrow layout. serde_arrow has no arrow dependency outside the conversions, so the 256 bit parsing and formatting is a small helper next to the existing decimal code;
  • Interval(MonthDayNano), mapped to jiff::Span strings as discussed in Add support for Interval types #178, plus structs with months, days, nanoseconds fields. A span has one sign, so intervals with mixed signs (BigQuery allows them) need the struct form. YearMonth and DayTime are left out to keep it small.

I also fixed the filed typo in the decimal builder error annotation, the builder is shared with Decimal256 now. It touches the same files as #331 in different places, I checked that it merges cleanly and the serde_arrow tests pass with it.

Full disclosure, I used AI to make these changes, in a responsible way and under my own specs and control. I do write my own code for a lot of things, but for features like this one, just adding new types and more mappings, it's easier to use AI nowadays. I'm not an antagonist of AI and use it responsibly.

Checklist

  • Run the pre-commit checks locally with uv run python x.py precommit
  • Add tests for any new functionality
  • Update the relevant changelog (Changes.md for serde_arrow, marrow/Changes.md for marrow)

@abdolence

Copy link
Copy Markdown
Author

The changes followed your style (both commits style and testing), however if you still don't like any AI usage, feel free to close the PR, I'll use different approach in my own client.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant