Repository navigation
Commit db5cfd0
Xiaowei Tan
feat: support releasing batches in RecordBatchMemoryCounter
Add uncount_batch, uncount_array, and uncount_batch_with_array_overhead
methods to RecordBatchMemoryCounter. The internal BufferIdSet is changed
to a reference-counted BufferIdMap so that a buffer's capacity is
subtracted from memory_usage only when its refcount drops to zero.
The per-type buffer walk is shared between counting and uncounting via
a visit_array_buffers visitor, avoiding code duplication.
Comprehensive test suite covers all acceptance criteria from #26140:
- Count/uncount round trip (batch, array, batch_with_array_overhead)
- Two-slice shared-buffer example from the issue
- View arrays sharing data buffers (StringView slices, BinaryView)
- Dictionaries sharing values across arrays
- All nested types: List, Map, Union, RunEndEncoded, FixedSizeList,
FixedSizeBinary, deeply nested (List<Struct<Dict>>)
- Slice-sharing for Union and RunEndEncoded
- Null bitmaps, empty arrays, empty batches, NullArray
- Binary/LargeBinary/Utf8/LargeUtf8 offset+data buffer pairs
- String slices sharing offset+data buffers
- ListView and LargeListView
- Boolean arrays (bit-packed)
- >16 distinct buffers with inline-to-hash promotion + removals
- count_batch_with_array_overhead shared across batches
- Double-uncount idempotency (no underflow)
- Randomized 200-op sequence across 10 diverse array types
(Int32, Int64, String, StringView, Float64, Dictionary, List,
Boolean, plus slices) verified against a reference model
- record_batch_memory benchmark with count_uncount case, no regression
Closes #261401 parent a23b89f commit db5cfd0
2 files changed
Lines changed: 1229 additions & 89 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
180 | 180 | | |
181 | 181 | | |
182 | 182 | | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
183 | 215 | | |
184 | 216 | | |
185 | 217 | | |
186 | 218 | | |
187 | 219 | | |
188 | | - | |
| 220 | + | |
| 221 | + | |
189 | 222 | | |
190 | 223 | | |
0 commit comments