multi-value slots

[rizar]

Thanks for releasing code!

I'm trying to understand how this implementation handles multi-value slot values introduced in MultiWOZ 2.1 (such as e.g. cheap|moderate). Do you render them in dialog summaries as is? At evaluation time, does the model have to output the correct multi-value literal for the output to be considered correct? Or is it enough to output just one possible value? I tried to read the code, but got a little lost. I would greatly appreciate your help!

[jshin49]

Sorry for the late reply. The notification got lost somehow.

Do you render them in dialog summaries as is?

For the training and dev portion, we skip the turn entirely.

At evaluation time, does the model have to output the correct multi-value literal for the output to be considered correct?

For test, we just include them and the correct dialogue state output should be cheap|moderate. Hence, our model will probably get it wrong.

Here are the references:

ds2/ds2/datasets/data_loader.py

Lines 96 to 97 in 80aa473

	if dataset in {"train", "dev"} and has_or_character(slot_values):
	continue

ds2/ds2/utils/fix_label.py

Lines 70 to 71 in 80aa473

	def has_or_character(slot_values: Dict[str, str]):
	return any('|' in v for v in slot_values.values())