cksum/hashsum: Support for non-UTF-8 input in checksum files by RenjiSann · Pull Request #6793 · uutils/coreutils

RenjiSann · 2024-10-18T13:59:40Z

Second part of splitting #6603.

This PR adds support for non-UTF-8 checksum file content for cksum and hashsum validation tools.

In practice, given a CHECKSUM file with the following content, non-UTF-8 characters can appear in 2 places:

in the filename
in a comment
All the other places are either algorithm name, which should be ASCII, or hexa/base64 digest, which should be ASCII as well.

$ cat CHECKSUM
SHA1 (filename-might-have-non-utf-8) = 058ab38dd3603703b3a7063cf95dc51a4286b6fe
# comment-might-have-non-utf-8

The most important step is to get rid of the call to BufRead::lines() to iterate on the lines of the CHECKSUM file, otherwise, it panics on non-UTF-8 characters.

This change makes heavy use of String/OsString/Vec<u8> conversion, which happen to not be equally treated on all platforms.

github-actions · 2024-10-18T14:25:45Z

GNU testsuite comparison:

Congrats! The gnu test tests/timeout/timeout is no longer failing!

github-actions · 2024-10-19T00:05:59Z

GNU testsuite comparison:

Skip an intermittent issue tests/tail/inotify-dir-recreate (fails in this run but passes in the 'main' branch)

RenjiSann · 2024-10-19T15:46:38Z

I closed the PR by mistake, wanted to mark it as ready :/

github-actions · 2024-10-19T16:12:00Z

GNU testsuite comparison:

Congrats! The gnu test tests/timeout/timeout is no longer failing!

RenjiSann · 2024-10-19T23:25:04Z

@cakebaker @BenWiederhake
PR is ready for review if one of you can take a look :)

github-actions · 2024-10-19T23:49:09Z

GNU testsuite comparison:

Congrats! The gnu test tests/tail/assert is no longer failing!
Skipping an intermittent issue tests/tail/inotify-dir-recreate (passes in this run but fails in the 'main' branch)

cakebaker · 2024-10-23T07:19:28Z

@RenjiSann sorry, there are some conflicts :|

RenjiSann · 2024-10-23T09:13:02Z

@RenjiSann sorry, there are some conflicts :|

Fixed ! 😁

github-actions · 2024-10-23T09:38:23Z

GNU testsuite comparison:

Skip an intermittent issue tests/tail/inotify-dir-recreate (fails in this run but passes in the 'main' branch)
Congrats! The gnu test tests/misc/usage_vs_getopt is no longer failing!

src/uucore/src/lib/features/checksum.rs

tests/by-util/test_cksum.rs

github-actions · 2024-10-23T13:02:33Z

GNU testsuite comparison:

Congrats! The gnu test tests/misc/usage_vs_getopt is no longer failing!

src/uucore/src/lib/lib.rs

github-actions · 2024-10-23T15:31:27Z

GNU testsuite comparison:

Congrats! The gnu test tests/misc/usage_vs_getopt is no longer failing!
Congrats! The gnu test tests/timeout/timeout is no longer failing!

src/uucore/src/lib/features/checksum.rs

github-actions · 2024-10-23T22:01:38Z

GNU testsuite comparison:

Congrats! The gnu test tests/misc/usage_vs_getopt is no longer failing!
Congrats! The gnu test tests/timeout/timeout is no longer failing!

github-actions · 2024-10-24T08:42:11Z

GNU testsuite comparison:

Congrats! The gnu test tests/timeout/timeout is no longer failing!

The full line will be matched against the selected regex anyway

github-actions · 2024-10-24T10:10:58Z

GNU testsuite comparison:

Skip an intermittent issue tests/timeout/timeout (fails in this run but passes in the 'main' branch)

RenjiSann · 2024-10-24T10:58:45Z

Ready for what should be a final review :)

cakebaker · 2024-10-24T12:58:48Z

Thanks :)

RenjiSann changed the title ~~cksum/hashsum: Support for non-UTF-8 inputs~~ cksum/hashsum: Support for non-UTF-8 input in checksum files Oct 18, 2024

RenjiSann force-pushed the checksum-utf8 branch from febab10 to 71bc35b Compare October 18, 2024 23:34

RenjiSann mentioned this pull request Oct 18, 2024

cksum: Accept non UTF-8 inputs #6603

Closed

RenjiSann force-pushed the checksum-utf8 branch from 71bc35b to 7fa39e4 Compare October 18, 2024 23:39

RenjiSann force-pushed the checksum-utf8 branch 3 times, most recently from 050ad32 to 91c1cd8 Compare October 19, 2024 15:45

RenjiSann closed this Oct 19, 2024

RenjiSann reopened this Oct 19, 2024

RenjiSann marked this pull request as ready for review October 19, 2024 15:46

RenjiSann force-pushed the checksum-utf8 branch from 91c1cd8 to 57ac761 Compare October 19, 2024 16:11

RenjiSann force-pushed the checksum-utf8 branch 3 times, most recently from 3285e74 to 953fa37 Compare October 19, 2024 23:23

RenjiSann force-pushed the checksum-utf8 branch from 953fa37 to 838d433 Compare October 23, 2024 09:12

RenjiSann force-pushed the checksum-utf8 branch from 838d433 to 61ca177 Compare October 23, 2024 12:36

sylvestre reviewed Oct 23, 2024

View reviewed changes

src/uucore/src/lib/features/checksum.rs Outdated Show resolved Hide resolved

cakebaker reviewed Oct 23, 2024

View reviewed changes

tests/by-util/test_cksum.rs Outdated Show resolved Hide resolved

cakebaker reviewed Oct 23, 2024

View reviewed changes

src/uucore/src/lib/lib.rs Outdated Show resolved Hide resolved

RenjiSann force-pushed the checksum-utf8 branch from 61ca177 to 7e45735 Compare October 23, 2024 14:43

RenjiSann force-pushed the checksum-utf8 branch 3 times, most recently from 23d0c71 to ebe53e0 Compare October 23, 2024 15:05

cakebaker reviewed Oct 23, 2024

View reviewed changes

src/uucore/src/lib/features/checksum.rs Outdated Show resolved Hide resolved

cakebaker reviewed Oct 23, 2024

View reviewed changes

src/uucore/src/lib/features/checksum.rs Outdated Show resolved Hide resolved

RenjiSann force-pushed the checksum-utf8 branch 3 times, most recently from 178cc03 to 50bfa3d Compare October 23, 2024 21:35

RenjiSann added 6 commits October 24, 2024 11:41

uucore(lib): add u8/OsString conversion utils

3031874

uucore(checksum): determine regex from full line, not trimmed

97b81a6

The full line will be matched against the selected regex anyway

test(cksum): add test for trailing spaces making an error

1c41a12

uucore(checksum): handle non-utf-8 text in checksum files

8490461

uucore(checksum): fix non-utf-8 filename printing

99cddd1

test(cksum): add tests for non-utf-8 handling in checksum files

6030ad9

RenjiSann force-pushed the checksum-utf8 branch from 463c959 to 6030ad9 Compare October 24, 2024 09:44

cakebaker merged commit cb8711f into uutils:main Oct 24, 2024

RenjiSann deleted the checksum-utf8 branch October 24, 2024 13:48

RenjiSann mentioned this pull request Oct 25, 2024

checksum: prepare further behavior fix with a rework #6822

Merged

BrewTestBot mentioned this pull request Nov 16, 2024

uutils-coreutils 0.0.28 Homebrew/homebrew-core#197947

Merged

RenjiSann mentioned this pull request Nov 17, 2024

cksum: --check can't handle non-UTF-8 in comments #6573

Closed

moonfruit mentioned this pull request Nov 22, 2024

uutils-selected 0.0.28 moonfruit/homebrew-tap#178

Closed

Uh oh!

Conversation

RenjiSann commented Oct 18, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Oct 18, 2024

Uh oh!

github-actions bot commented Oct 19, 2024

Uh oh!

RenjiSann commented Oct 19, 2024

Uh oh!

github-actions bot commented Oct 19, 2024

Uh oh!

RenjiSann commented Oct 19, 2024

Uh oh!

github-actions bot commented Oct 19, 2024

Uh oh!

cakebaker commented Oct 23, 2024

Uh oh!

RenjiSann commented Oct 23, 2024

Uh oh!

github-actions bot commented Oct 23, 2024

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 23, 2024

Uh oh!

Uh oh!

github-actions bot commented Oct 23, 2024

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Oct 23, 2024

Uh oh!

github-actions bot commented Oct 24, 2024

Uh oh!

github-actions bot commented Oct 24, 2024

Uh oh!

RenjiSann commented Oct 24, 2024

Uh oh!

cakebaker commented Oct 24, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

RenjiSann commented Oct 18, 2024 •

edited

Loading