fstree: never copy to io.Discard unless dealing with compression#4016
Open
roman-khimov wants to merge 1 commit into
Open
fstree: never copy to io.Discard unless dealing with compression#4016roman-khimov wants to merge 1 commit into
roman-khimov wants to merge 1 commit into
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4016 +/- ##
==========================================
- Coverage 27.79% 27.75% -0.04%
==========================================
Files 681 681
Lines 46797 46942 +145
==========================================
+ Hits 13006 13031 +25
- Misses 32553 32669 +116
- Partials 1238 1242 +4 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR refactors FSTree read/range-stream handling to avoid falling back to io.CopyN(io.Discard, ...) when skipping data for non-compressed objects, by switching internal readers to seek-capable wrappers.
Changes:
- Introduces seek-capable reader wrappers (
prefixedReadSeekCloser,limitedFileReader) and a zstdcompressedReader. - Changes internal object reading helpers to return
io.ReadSeekCloserto enable skipping viaSeek. - Updates payload prefix/range handling to use the new wrappers instead of
io.MultiReader+io.LimitReadercompositions.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| pkg/local_object_storage/blobstor/fstree/util.go | Adds new reader wrapper types used to prefix/limit streams and handle compressed reads. |
| pkg/local_object_storage/blobstor/fstree/head.go | Switches internal read paths to return seek-capable readers and uses new prefix wrapper. |
| pkg/local_object_storage/blobstor/fstree/fstree.go | Updates range-shifting logic to use Seek and new limiting/prefix wrappers. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| @@ -579,7 +579,7 @@ func (t *FSTree) readPayloadRange(addr oid.Address, off, ln uint64, getHdrBuf fu | |||
| return pldLen, stream, nil | |||
Member
Author
There was a problem hiding this comment.
True, but I don't want any additional wrappers. Current users are well known.
FSTree routinely creates io.MultiReader in many places and the probelm is that it does not and can not implement Seek(), so every time we're trying to get a range stream we're effectively reading and discarding everything which is not very productive. This patch introduces a new set of wrappers with Seek() implemented. These implementations are not complete, it's more of Skip() rather than Seek(), but that's the only thing we really need and this allows to reuse the same io.ReadSeekCloser everywhere (implemented natively by os.File). Signed-off-by: Roman Khimov <roman@nspcc.ru>
roman-khimov
force-pushed
the
fstree-ranges
branch
from
54ebd39 to
71faf77
Compare
roman-khimov
requested review from
End-rey,
carpawell and
cthulhu-rider
as code owners
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This suggests compression is harmful, so it better be dropped. But that's subject to some subsequent PR.