Request for UTF-8 support

[joakim-tjernlund]

Any idea if UTF support can be added to this lib?
Preferably without depending on locale stuff from glibc

[troglobit]

Sure it can, it's just a lot of work since most of the lib is built around the notion that a character is one byte wide. A Unicode character, as far as I know, can be two, three, or four bytes wide.

Do you mind telling med the usecase? If our interests interlock (which I've noticed quite a few times) I may be able to help in such an endeavor.

[joakim-tjernlund]

Today most terminals default to UTF-8 so i think it wold be great if editline could do UTF-8
but limit accepted chars to ISO Latin .
That would help our use case where users can connect via telnet/ssh/rs232 and non ascii
chars would just work.

[troglobit]

Yeah, that's a good limitation. I'll have a look again and try to estimate the amount of work needed.

[troglobit]

Interestingly quite a lot actually works already. What needs attention is cursor movement, most notably when editing a line. It is hard-coded to one-character=one-byte ... also, most of the internals use 'int' to store a character during processing. So it shouldn't be too much work to straighten out libeditline \o/

[joakim-tjernlund]

Any progress on this?

[troglobit]

Nope. I've been redirected/reprioritized at work atm, so I'll have to circulate back to this later. Sorry.

[skull-squadron]

When there is time, the easiest way to do this would be like how another library handled it: basically duplicate the API (functions, header and library) and append a suffix such as w to all relevant symbols. Then, use something like libunistring, utf8proc or icu4c to handle conversion from raw bytes to UTF-8, and then iterating on individual 21-bit code points (stored 1 to 4 bytes long). There are numerous edge-cases like non-/UTF-8 BOMs to discard at the beginning of input, code-points spanning reads and UTF-16 surrogate pairs to (not) handle, so writing a solid parser yourself isn't easy. It gets even more complicated (converting the latest Unicode standard version into optimal conditional statements) to decide what 21-bit value is valid Unicode.

[troglobit]

@steakknife Thank you for the input! I sort of came to the same conclusion wrt parsing, but I'm really hesitant to make editline depend on other libraries, the whole point of editline is to be small and have as few (no) dependencies as possible. Whenever I circle back to this (or someone else does), your input will be a valuable starting point for further investigation!

[Artoria2e5]

I doubt there is any need for an extra layer on top of that, let along any libraries. The one thing we need to get from a UTF-8 string is the width of characters. Every POSIX system has a int wcwidth(wchar_t c), and all we need is to use mbrtowc to feed it. (On Windows you will need to carry your own wcwidth table and the conversion function. busybox has a nice slim version.)

[troglobit]

Good point. Just remember that BusyBox is GPL, this lib isn't.

[minfrin]

Replxx (BSD license) has an example of calculating the number of unicode codepoints in a UTF8 string:

https://github.com/AmokHuginnsson/replxx/blob/master/examples/util.c#L3

[timkuijsten]

When it comes to supporting different locales I found this presentation by the OpenBSD maintainer of libedit very helpful: https://czyborra.com/yudit/eurobsdcon2016-utf8.pdf

Supporting UTF-8 might be possible without having to duplicate all interfaces or dragging in heavy weights like libiconv.

[skull-squadron]

@troglobit To be honest, Unicode (UTF-8, -16LE/BE, -32LE/BE, and UCS2/4) support including BOM detection/removal, sanitization, and normalization is trivial-adjacent. The catch is 95% of libraries get it wrong and/or don't keep it current, including tools at work used to parse Rust code. (We're in the midst of a refactoring bonanza.) The maintenance required is precision around keeping current with Unicode versions and correctly classifying characters that could fall under multiple domains. The older Unicode references list these concerns more pedantically than newer ones. It's also worth being away of runtime behavioral changes imposed by LC_CTYPE, LC_COLLATE, LC_ALL, and LANG in GNU/Linux, POSIX, and other systems. For example, when grepping random binary files for text, better set LC_ALL=C or it's likely to assume UTF-8 is the lingua franca.