dd: optimize O_DIRECT buffer alignment to reduce syscall overhead#9104
Open
naoNao89 wants to merge 7 commits intouutils:mainfrom
Open
dd: optimize O_DIRECT buffer alignment to reduce syscall overhead#9104naoNao89 wants to merge 7 commits intouutils:mainfrom
naoNao89 wants to merge 7 commits intouutils:mainfrom
Conversation
9f131bd to
fecebff
Compare
CodSpeed Performance ReportMerging #9104 will not alter performanceComparing Summary
Footnotes
|
Implement page-aligned buffer allocation and optimize O_DIRECT flag handling to match GNU dd behavior. Key changes: - Add allocate_aligned_buffer() for page-aligned memory allocation - Update buffer allocation to use aligned buffers - Modify handle_o_direct_write() to only remove O_DIRECT for partial blocks - Add Output::write_with_o_direct_handling() for proper O_DIRECT handling - Add comprehensive unit and integration tests Fixes uutils#6078
fecebff to
2560240
Compare
…IRECT on ARM O_DIRECT requires page-aligned buffers and writes. The conv=sync flag pads output to block size, which may not be page-aligned, causing EINVAL errors on ARM systems. The core O_DIRECT functionality is already well-tested by: - test_o_direct_with_aligned_buffer_full_blocks - test_o_direct_with_partial_final_block - test_o_direct_various_block_sizes
|
GNU testsuite comparison: |
Contributor
Author
|
I need more dopamine when stuck on a bug, so new PRs might be good :)) |
|
GNU testsuite comparison: |
|
GNU testsuite comparison: |
cre4ture
reviewed
Nov 17, 2025
| /// This function allocates a `Vec<u8>` with proper alignment to support O_DIRECT | ||
| /// without triggering EINVAL errors. | ||
| #[cfg(any(target_os = "linux", target_os = "android"))] | ||
| fn allocate_aligned_buffer(size: usize) -> Vec<u8> { |
Contributor
There was a problem hiding this comment.
@sylvestre is this something we could move to a more central location? Or is this the only place where we need aligned memory allocations?
Contributor
There was a problem hiding this comment.
which programs will use it ? thanks
- Remove dead code: non-Linux stub for handle_o_direct_write The stub was unreachable since write_with_o_direct_handling already has a non-Linux stub that doesn't call this helper function. - Fix clippy::ptr-as-ptr lint error Replace unsafe `as *mut u8` cast with safer `.cast::<u8>()` method in allocate_aligned_buffer function. Addresses review comments and CI/CD failures in PR uutils#9104.
|
GNU testsuite comparison: |
Removed redundant buffer initialization in allocate_aligned_buffer that was causing performance regression, especially for large block sizes. - Eliminated O(n) write_bytes overhead that scaled with buffer size - Fixes 29.36% regression for 1M blocks and 6.22% for 64K blocks - Buffer is correctly filled during copy operations, making pre-init redundant
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #6078
page-aligned buffers + smarter O_DIRECT handling. Theory says 5x fewer syscalls. 🗿
Checklist: