Replacing QTextCodec

[WickedSmoke]

QTextCodec requires the Core5Compat library with Qt6. The replacement seems to be QStringConverter, but there is no equivalent of QTextCodec setCodecForLocale()/codecForLocale().

What is the purpose of the "Text codec" Setting? Shouldn't this automatically be set to whatever encoding is used in the repository rather than being a user option?

Why is there a hardcoded list of encodings (const char* en[]) in settingsimpl.cpp?

[tibirna]

This feature was coded into QGit before I took up maintenance and there are no design documents to go consult.

My intuition is that the "Text codec" setting is meant to cope with the ability of users to produce qgit repositories with one local codec (e.g. the awfully sticky windows cp1252) on one platform, and having to port that to some other, otherwise incompatible platform (e.g. Linux used utf-8 for everything for at least 20 years).

Character encodings are notoriously difficult to detect automatically, so manual settings are still seen in most text-processing tools. Git itself has encoding settings https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration.

The hardcoded list of encoding descriptions is there solely for the benefit of filling the "Text Codec" combo box, perhaps because at the time the list wasn't retrievable from Qt.

And unfortunately I didn't investigate Qt's decision to degrade support for character encodings going on from Qt5 to Qt6,. I can only guess that the sentiment was that enough time passed since the Babel tower times of quasi-infinite codec lists, and that everybody coverges to some UTF variant (preferably UTF-8, although, for practical reasons, both Qt and Windows use UTF-16 internally).

I would very much prefer to get rid of this headsplitting complication, but I already had QGit users that complained of bugs in handling various encodings (mostly the russian koi8-R one), especially for supporting old repositories, created in the time of the codec wild west (git is 20 years old!).

Thanks for keeping the topic alive. This issue needs a solid and sound solution. It requires proper investigation and a careful resolution plan and execution. This unfortunately means non-trivial time investment, which I won't have for the next few months.

[WickedSmoke]

Yeah, I saw that code in the first commit. Is there a repository for older versions of QGit?

The git config gui.encoding sets a default file encoding. The i18n.commitEncoding used by QGit looks to be for the commit messages only, so that may being misused.

[tibirna]

There is a copy of an old CVS repository here: https://sourceforge.net/code-snapshots/cvs/q/qg/qgit.zip. And several old releases (code only, no changes history) here: https://sourceforge.net/projects/qgit/files/qgit/

QGit is only concerned with the encoding of the commit messages because it has to display them properly (by applying the requested encoding to the raw data extracted from the repo before writing it to the text widgets etc.). The rest (file paths, mostly) is handled by the underlying git commands.

[WickedSmoke]

I have some working code which uses QStringConverter and also adds an extra combo for the gui.encoding. I would have thought the content encoding needed to be handled in the file view & diff output.

[tibirna]

It appears your before last comment, containing an URL to another repo federator, is not displayed by github. But I was able to see your modifications and things look great! Thanks a lot for the effort!

One thing qgit misses dearly is test cases. The encoding handling is one of the things that would be best served by some example repositories to test against.

[WickedSmoke]

I deleted that post since I'm not certain about the role of gui.encoding yet. I should probably break that commit into two by adding the QStringConverter first and then the extra control.

[WickedSmoke]

A brief search for test data has only turned up a small repo, but that's enough to confirm that QGit is really lacking here.

On my dev branch I just made FileContent aware of gui.encoding by changing a line of code.

[WickedSmoke]

I looked at what "git gui" and gitk are doing with gui.encoding and it matches my understanding of the documentation. I don't see any way to change i18n.commitEncoding in "git gui".

I'll move ahead with the two separate controls (and repost the screenshot).

As these modify the .git/config a case could be made that the controls belong in the "Git config" tab.

[WickedSmoke]

The Git documentation says this:

Commit log messages are typically encoded in UTF-8, but other extended ASCII encodings are also supported. This includes ISO-8859-x, CP125x and many others, but not UTF-16/32, EBCDIC and CJK multi-byte encodings (GBK, Shift-JIS, Big5, EUC-x, CP9xx etc.).

Those CJK encodings are all in the hardcoded en[] array and so maybe they shouldn't be there.

[tibirna]

The Git documentation says this:

Commit log messages are typically encoded in UTF-8, but other extended ASCII encodings are also supported. This includes ISO-8859-x, CP125x and many others, but not UTF-16/32, EBCDIC and CJK multi-byte encodings (GBK, Shift-JIS, Big5, EUC-x, CP9xx etc.).

Those CJK encodings are all in the hardcoded en[] array and so maybe they shouldn't be there.

Well, the git documentation only says what the git executable does. It most certainly pertains to displaying data in the terminal. Otherwise, that data is simply streams of bytes, and git won't modify those (although it does do modifications for line ending markers, for example), whichever encoding they come from. If some exotic codec was used, git might not display it correctly, but it won't corrupt it and other tools can well handle it properly. QGit probably chose to support all those other codecs, just in case, because it could (at the time), with help from Qt's text encoding mechanisms.

All that just to say that it's not necessary that QGit matches git in this behaviour, but it's of course a possible choice.