MultiWOZ 2.2, 2.3 데이터셋을 사용할 수 있도록 코드 수정.#4
Open
JJinIT wants to merge 5 commits intoclovaai:masterfrom
Open
Conversation
mwz_2.2, 2.3 adding
Add MultiWOZ 2.2 and 2.3 dataset
Revert "Add MultiWOZ 2.2 and 2.3 dataset"
|
@JJinIT 안녕하세요. 먼저 converting된 2.2와 2.3 데이터셋을 업로드해주셔서 감사합니다. 2.2 데이터셋을 확인해보니 ontology 값이 MultiWOZ의 2.2 schema.json과 다른 것 같습니다. (현재 업로드된 Ontology는 2.1 version으로 생각됩니다) 아래의 링크에 schema.json을 MultiWOZ 2.1 format으로 변경한 ontology.json 파일을 업로드 해두었습니다. 감사합니다 🤗 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
som-dst모델에서 기본적으로 MultiWOZ 2.1을 사용하고 있는데요.
최근에 2.2와 2.3이 릴리즈되어서 최신 데이터셋도 사용할 수 있도록 해보았습니다.
데이터셋은 2.1 포맷으로 컨버팅하여 JJinIT/som-dst에 wget으로 다운로드 가능하게 해두었구요.
개인적으로 테스트해보니 2.2일떄 조금 더 성능이 잘 나오는것 같네요.
2.3에서는 성능이 현저하게 떨어지는데, 저희쪽 실험 환경이 잘못되었을 수 있습니다.
아래는 Google Colab에서
batch_size=16으로 30 epochs 돌리고 얻은 evaluation 결과들입니다.2.1 (baseline)

2.2 (https://github.com/budzianowski/multiwoz/tree/master/data/MultiWOZ_2.2)

2.3 (https://github.com/lexmen318/MultiWOZ-coref)

cc. @d4rk6un