KGSF 테스트 시 배치마다 입력 데이터와 evaluation 결과 출력하는 기능 추가#2
Open
krta2 wants to merge 23 commits intoadd-kgsf-test-and-interactfrom
Open
KGSF 테스트 시 배치마다 입력 데이터와 evaluation 결과 출력하는 기능 추가#2krta2 wants to merge 23 commits intoadd-kgsf-test-and-interactfrom
krta2 wants to merge 23 commits intoadd-kgsf-test-and-interactfrom
Conversation
Add test_print_every_batch option
- add print every batch feature
- add print every batch in rec test
krta2
changed the title
* Update KGSFSystem - add test method * Update QuickStart - add test for restore_system option Co-authored-by: airotod <airotod@gmail.com>
* Update kgsf/redial config - add test_print_every_batch option for conv * Update BaseDataLoader - add print every batch feature of conv * Update KGSFSystem - add print every batch in conv test - add print conv response and prediction Co-authored-by: airotod <airotod@gmail.com>
- remove print every batch feature in train
- test option enable restore_system automatically
krta2
changed the title
print conversation prediction/response only when test_print_every_batch
krta2
force-pushed
the
add-print-every-batch
branch
from
8d6615e to
0f91563
Compare
krta2
changed the base branch from
main
to
add-print-batch-and-kgsf-interact
krta2
changed the base branch from
add-print-batch-and-kgsf-interact
to
main
* Update Config - add path for saving csv * Update BaseDataLoader - add write sentence into csv file * Update StandardEvaluator - add write metric into csv file * Update KGSFSystem - add write test result - add write prediction and response for conv into csv file * Update Config - update path for saving csv * Update BaseDataLoader - not log when writing csv file * Update StandardEvaluator - not log when writing csv file * Update KGSFSystem - update csv path variable name - not log when writing csv file - remove unnecessary comment
- Open file only when needed
- print file only in test mode (--test) and test_print_every_batch flag on.
- automatically make file output dir when not exist
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
KGSFSystem.train_recommender의 테스트 단계에서 배치별로 입력 context 에 대한 데이터와 해당 배치의 성능 평가 결과를 파일로 출력해 어떤 경우에 안 좋은 성능이 나오는지 분석할 수 있도록 합니다.참고: https://www.notion.so/krta2/Redial-d49644b8e869487187166e130975f708
How to run
python run_crslab.py -c config/crs/kgsf/redial.yaml -t -g 0
TODO