Math bug in latin1prober

[ablegrape]

(dcramer/erikrose - first, thanks for taking an interest in chardet - good to see someone is rescuing this useful package from oblivion).

Just noticed a serious logic bug in latin1prober. At line 133, you'll see:

 confidence = (self._mFreqCounter[3] / total) - (self._mFreqCounter[1] * 20.0 / total)

The problem with this is that self._mFreqCounter[3] and total are both integers, so this first term is always 0 any time self._mFreqCounter[3] != total, meaning that confidence is 0 for all cases where self._mFreqCounter[3] != total. In other words, even one "unlikely" character transition in a document of any length will produce a confidence of 0. This is certainly not the behavior of the original Mozilla code, which does this all in floating point.

An example document which shows this problem can be found at: http://www.lvo.com/GASTRONOMIE/VINS/VITI/VITI1F.HTML

A simple fix would be to change this line of code to:

 confidence = (self._mFreqCounter[3] - self._mFreqCounter[1] * 20.0) / total

which obviates the multiple divisions as well.

As an aside, the "0.5" confidence multiplier (a few lines later in the code) is a wild guess (as the original authors note, this is sort of a hacky Latin-1 detector anyway). I've had better experience with 0.8 (for example, the document given in this report is incorrectly detected as 'iso-8859-2,' which wreaks havoc on the accents, until about 0.8), though your mileage may vary.

[dcramer]

Some good info. To clarify, I believe I took down chardet from the pypi version (though I dont recall anymore) and modified it from there.

Most of my changes were bound to certain things like removing unsafe conversions, and fixing several obvious errors. It sounds like these may have been introduced through some changes outside of yourself at some point though.

[ablegrape]

Hi David,

Thanks for the quick response.

I posted the issue here since this seems to be the most active development on chardet - Mark Pilgrim's homepage link on pypi is long broken, and erikrose seems to have put his 1.1 version, from this github source, up on pypi yesterday.

The bug I'm mentioning has probably been present since Mark Pilgrim's original port; it's in the original pypi version (1.0.1) and in the new 1.1 version, so it's definitely not new and not specific to anything you or Erik has done.

I had thought it might be easier for someone already doing development on chardet to integrate this fix, but I'll gladly fork a repo of my own and put this fix in it if that's easier. I'm fairly new to github, but for learning new things there's no time like the present.

[Ousret]

Hi, I gave a shot at trying to propose a viable alternative to chardet. https://github.com/Ousret/charset_normalizer
It's covering around 90 encodings.