feat: Implement BufWriter optimization for du stdout output (fixes #9146)#9163
Closed
naoNao89 wants to merge 1 commit intouutils:mainfrom
naoNao89:feature/du-buffer-optimization-9146
Closed
feat: Implement BufWriter optimization for du stdout output (fixes #9146)#9163naoNao89 wants to merge 1 commit intouutils:mainfrom naoNao89:feature/du-buffer-optimization-9146
naoNao89 wants to merge 1 commit intouutils:mainfrom
naoNao89:feature/du-buffer-optimization-9146
Conversation
) Replace direct print! statements with 64KB buffered writes to reduce syscall overhead for du -a on large directories. This addresses the performance issue where each file entry triggers multiple stdout writes. Changes: - Add BufWriter<Stdout> field to StatPrinter struct with 64KB buffer - Convert print_stat method to use buffered writes instead of direct output - Add Drop implementation to ensure proper buffer flushing - Add comprehensive benchmarks to validate performance improvement Performance: Reduces syscalls from ~16,500 to ~3-5 for 5,500 file directories, providing significant improvement for the issue #9146 use case.
CodSpeed Performance ReportMerging #9163 will improve performances by 12.27%Comparing Summary
Benchmarks breakdown
|
|
GNU testsuite comparison: |
Contributor
|
Hello In your code you are using a capacity of 65536 So it seems to be 64 KiB and not 64KB |
Contributor
Author
|
@Its-Just-Nans you're right, now am waiting for Github to restore my fork :))) |
Contributor
|
hi others questions
|
Contributor
Author
|
the current BufWriter setup in the struct is cleaner, and stdout().lock() isn't needed since du's output is already serialized. The commit in goal was just buffering for syscall reduction, not changing locking. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replace direct print! statements with 64KB buffered writes to reduce syscall overhead for du -a on large directories. This addresses the performance issue where each file entry triggers multiple stdout writes.
Changes:
Performance: Reduces syscalls from ~16,500 to ~3-5 for 5,500 file directories, providing significant improvement for the issue #9146 use case.