How to use Unidic for Japanese?

[patarapolw]

mecab-python3 itself doesn't recommend ipadic anymore.

In order to use MeCab, you must install a dictionary. There are many different dictionaries available for MeCab. These UniDic packages, which include slight modifications for ease of use, are recommended:

• unidic: The latest full UniDic.
• unidic-lite: A slightly modified UniDic 2.1.2, chosen for its small size.

The dictionaries below are not recommended due to being unmaintained for many years, but they are available for use with legacy applications.

• ipadic
• jumandic

For more details on the differences between dictionaries see here.

Furthermore, other tokenizers might also be considered. (but a little out of scope, and can create more confusion, perhaps.)

[rspeer]

I recognize that ipadic is old and busted, but unfortunately, using Unidic would cause the tokenization not to match the word frequencies that were extracted from the source corpora. We would have to re-compute the frequencies.

Also unfortunately, one of the source corpora belongs to my old company and I have no access to it.