mark_unknown doesn't work

[dimitarsh1]

Hi,
when translating, setting mark_unknown to False, does not impact the translation at all, thus always placing a "*" in front of unknown and "@", "#" and "/" in front of errors.

Furthermore, in the translate function in __init__.py it seem that the mark_unknown argument does not do anything; it is not invoked or used anywhere.

Any idea how to fix this?

Thanks in advance,
Dimitar

[sushain97]

Yeah, this is a bug.

[dimitarsh1]

Yes, I see. Thank you.

…

On Wed, 6 May 2020, 07:01 Sushain Cherivirala, ***@***.***> wrote: Yeah, this is a bug. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#82 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADYLMETW33LIJGA2NREENLTRQD4NDANCNFSM4M2ER4UQ> .

[dimitarsh1]

OK,

so, I took some time to get what's going on in the code and what calls what. In the lttoolbox there is no support for unknown.

However, I did a quick fix the following way:
In utils.py in handle_command_with_wrapper:

` ....
text = end.decode()
input_file.write(text)
input_file.close()

if 'lt-proc' == command[0]:
    fst = initialized_wrappers[command]
    lt_proc_command, dictionary_path, arg = command[:-1], command[-1], command[1]
    fst.lt_proc(lt_proc_command, input_file.name, output_file.name)

`

replaced with
`
text = end.decode()
input_file.write(text)
input_file.close()

-->if 'lt-proc' == command[0] and "-n" != command[1]:
fst = initialized_wrappers[command]
lt_proc_command, dictionary_path, arg = command[:-1], command[-1], command[1]
fst.lt_proc(lt_proc_command, input_file.name, output_file.name)
`

Then changed also parse_mode_file:

from
def parse_mode_file(mode_path: str) -> List[List[str]]: """ .... cmd = cmd.replace('$2', '').replace('$1', '-g') ....
to
def parse_mode_file(mode_path: str, mark_unknown: bool = True) -> List[List[str]]: .... if not mark_unknown: cmd = cmd.replace('$2', '').replace('$1', '-n') else: cmd = cmd.replace('$2', '').replace('$1', '-g') ....

Then in the translate/__init__.py I changed _get_commands:

`def _get_commands(self, l1: str, l2: str, mark_unknown: bool = True) -> List[List[str]]:
"""
Args:
l1 (str)
l2 (str)

    Returns:
        List[List[str]]
    """
    if (l1, l2) not in self.translation_cmds:
        mode_path = apertium.pairs['%s-%s' % (l1, l2)]
        self.translation_cmds[(l1, l2)] = parse_mode_file(mode_path, mark_unknown)
    return self.translation_cmds[(l1, l2)]

`

and Translator.translate:
cmds = list(self._get_commands(l1, l2)) to cmds = list(self._get_commands(l1, l2, mark_unknown))

Also, the default value for mark_unknown everywhere is set to "True".

Don't know if that's a good fix - haven't had the time to delve into lttoolbox and FST, but it seems to work for me.
System is ubuntu 18.04; python is 3.6.5

Kind regards,
Dimitar

[sushain97]

Could you send a proper diff/patch or PR? Your comment is really hard to read.

[ygorg]

Reformatting the comment from @dimitarsh1 :
In the lttoolbox there is no support for unknown.

In

apertium-python/apertium/utils.py

Line 110 in 81b10e5

if 'lt-proc' == command[0]:

in handle_command_with_wrapper:

if 'lt-proc' == command[0] and "-n" != command[1]:

Then changed also parse_mode_file:

apertium-python/apertium/utils.py

Line 195 in 81b10e5

cmd = cmd.replace('$2', '').replace('$1', '-g')

to

# Add parameter mark_unknown
def parse_mode_file(mode_path: str, mark_unknown: bool = True) -> List[List[str]]


    if not mark_unknown:
        cmd = cmd.replace('$2', '').replace('$1', '-n')
    else:
        cmd = cmd.replace('$2', '').replace('$1', '-g')

Then in the translate/__init__.py I changed _get_commands:

apertium-python/apertium/translation/__init__.py

Line 38 in 81b10e5

self.translation_cmds[(lang1, lang2)] = parse_mode_file(mode_path)

# Add parameter mark_unknown
def _get_commands(self, l1: str, l2: str, mark_unknown: bool = True) -> List[List[str]]:

        self.translation_cmds[(l1, l2)] = parse_mode_file(mode_path, mark_unknown)

and Translator.translate:

apertium-python/apertium/translation/__init__.py

Line 168 in 81b10e5

cmds = list(self._get_commands(lang1, lang2))

cmds = list(self._get_commands(l1, l2, mark_unknown))

Also, the default value for mark_unknown everywhere is set to "True".