Enhancement: Properly handle encoding and decoding of raw non-UTF-8 binary data

[lcn2]

Is there an existing issue for this?

• I have searched for existing issues and did not find anything like this

Describe the enhancement

Now all sequences of bytes are valid UTF-8.

The JSON encoder and JSON decoder should handle raw non-UTF-8 binary data in some way.

Relevant images, screenshots or other files

From issue 1056 from mkiocccentry repo GH-issuecomment-2566736763 as reported by @SirWumpus:

elf$ jparse/jstrencode -v 0 -n < jparse/jstrencode | wc
wc: <stdin>: invalid byte sequence
wc: <stdin>: invalid byte sequence
wc: <stdin>: invalid byte sequence
wc: <stdin>: invalid byte sequence
wc: <stdin>: invalid byte sequence
wc: <stdin>: invalid byte sequence
wc: <stdin>: invalid byte sequence
wc: <stdin>: invalid byte sequence
wc: <stdin>: invalid byte sequence
wc: <stdin>: invalid byte sequence
wc: <stdin>: invalid byte sequence
wc: <stdin>: invalid byte sequence
wc: <stdin>: invalid byte sequence
wc: <stdin>: invalid byte sequence
...

Relevant links

• issue 1056 from mkiocccentry repo
• JSON Spec for strings GH-issuecomment-2567302558
• issue 1056 from mkiocccentry repo GH-issuecomment-2567305110

Anything else?

If raw non-UTF-8 binary data is not properly handled in some way, warnings can be issued by commands that object to invalid UTF-8:

• issue 1056 from mkiocccentry repo GH-issuecomment-2566736763

Or worse, programs might hang as a result of invalid UTF-8:

• issue 1056 from mkiocccentry repo GH-issuecomment-2566738203

Unless some mechanism, some form of warning/error handling and/or some form of encoding is used to protect code from invalid UTF-8.

[xexyl]

Thanks for opening this. It still seems weird that macOS and Linux have no problems but NetBSD does.

It's a baffling situation for me.

[xexyl]

See also issue #28.

But as far as this goes. The encode routines use the table you added ages ago.

I wonder if this code needs to be updated.

What do you think @lcn2 ?

I am not looking at the code and I really am not sure how to test this as I don't have a problematic system and that makes it hard to address.

I didn't see the problem in the files uploaded or the commands' output. But I also didn't read the full thread as I had the other issues over there to take care of. But I guess I might still have missed it.

Not sure.

[lcn2]

I wonder if this code needs to be updated.

Very likely.

We do recommend reading these 2 links:

• GH-issuecomment-2567302558

• GH-issuecomment-2567305110

And in particular this Wikipedia link:

• https://en.wikipedia.org/wiki/UTF-8#Error_handling

[lcn2]

Please consider this comment by @SirWumpus:

GH-issuecomment-2568247131

We encourage you, @xexyl, to look at his code use and if possible, incorporate his idea.

[SirWumpus]

Sidesaddle: @lcn2 his toll use odd turn of phrase in this context.

[lcn2]

Sidesaddle: @lcn2 his toll use odd turn of phrase in this context.

Corrected the comment.

[lcn2]

From GH-issuecomment-2569773830 in "the other repo':

More detailed test code; also broke up do/while loop test expression into more manageable parts. Does this help a little more? Maybe a man page? 🤔

utf8decode.c.txt utf8encode.c.txt

Thank you @SirWumpus

FYI: @xexyl

With that code, you can do this:

./utf8encode < ./utf8encode | ./utf8decode | cmp - ./utf8encode

This would allow the two tests that were previously disabled due to issue 1056 in "the other repo" and thus undo commit b54b827e4d4aaaaeea245d2341a44476f619cb1b in "the other repo".

We suggest that you use his algorithm / code to satisfy this issue.

Perhaps using his algorithm / code is the way to satisfy this issue?

See also GH-issuecomment-2569896762 in "the other repo",

Sidehussle: You'd think for a repository web site aiding in C code development you could upload .c files in comments without having to rename extensions.

Wow, you would think so.

[lcn2]

Thanks for opening this. It still seems weird that macOS and Linux have no problems but NetBSD does.

It's a baffling situation for me.

Our Guess

Our guess this is because the NetBSD code (perhaps deep in libc) is working "wide chars", and thus paying attention to things like UTF-8 encoding, a then "going bonkers" as the express goes when it encounters invalid sequences from the UTF-8 perspective.

Even so, the NetBSD wc(1) did highlight a problem that was worth noting.

As to the pipe hang, we can how someone could write a poor implementation of an encoder and/or decoder that could go into what seems like an infinite loop. For what I can see, @SirWumpus code does do that: perhaps because his algorithm is better, or perhaps because there is some tiny corner case his code doesn't handle, or maybe there is some corner case where decoding does not perfectly inverse encoding. There is also the possibility that some "wide char" code developer decided to become pedantic and have their code "throw a fit" rather than to mange binary non-UTF-8 stuff in a nice way.

My "bet" would be on that SirWumpus' code is just better.

[SirWumpus]

./utf8encode < ./utf8encode | ./utf8decode | cmp - ./utf8encode

From the other repo the above can be done with the single tool:

./utf8str < ./utf8str | ./utf8str -d | cmp - ./utf8str

[xexyl]

From GH-issuecomment-2569773830 in "the other repo':

More detailed test code; also broke up do/while loop test expression into more manageable parts. Does this help a little more? Maybe a man page? 🤔
utf8decode.c.txt utf8encode.c.txt

Thank you @SirWumpus

I'll look into this after syncing jparse to mkiocccentry and opening a pull request .. thanks @SirWumpus!

FYI: @xexyl

With that code, you can do this:

./utf8encode < ./utf8encode | ./utf8decode | cmp - ./utf8encode

Hopefully the algorithm can be integrated with ease .. or at least enough ease .. will look at it soon (I hope anyway).

This would allow the two tests that were previously disabled due to issue 1056 in "the other repo" and thus undo commit b54b827e4d4aaaaeea245d2341a44476f619cb1b in "the other repo".

Well I would first get it to compile and then add back these tests indeed to make sure it works out. That was my plan all along.

We suggest that you use his algorithm / code to satisfy this issue.

Perhaps using his algorithm / code is the way to satisfy this issue?

Hopefully that's the case. If so do you think we can get rid of that byte2asciistr struct and table?

See also GH-issuecomment-2569896762 in "the other repo",

I'll do that!