Support generation tasks and various improvements#3
Support generation tasks and various improvements#3ZhaofengWu wants to merge 3 commits intomsclar:mainfrom
Conversation
|
|
||
| # SuperNaturalInstructions Tasks without a defined format | ||
| if any(t in args.task_filename for t in SUPERNATURAL_INSTRUCTIONS_TASKS_WITH_NO_FORMAT): | ||
| if any(t in args.task_filename for t in SUPERNATURAL_INSTRUCTIONS_TASKS_WITH_NO_FORMAT + OPEN_GENERATION_SUPERNATURAL_INSTRUCTIONS_TASKS): |
There was a problem hiding this comment.
Seems like _setup_non_formatted_dataset_with_one_field_only works well for generation tasks out of the box.
| inputs = tokenizer(prompt_list, padding=True, return_tensors='pt', return_token_type_ids=False).to('cuda') | ||
| if tokenizer.chat_template is None: | ||
| inputs = tokenizer(prompt_list, padding=True, return_tensors='pt', return_token_type_ids=False).to('cuda') | ||
| else: |
|
|
||
| generated_answer_list = [s.lower() for s in tokenizer.batch_decode(outputs['sequences'], skip_special_tokens=True)] | ||
| sequences = outputs['sequences'] | ||
| if tokenizer.chat_template is not None: |
There was a problem hiding this comment.
The truncation of chat models is a bit different from regular models and need to be done separately. Also got rid of lowercasing for generation tasks (done below for classification tasks).
| full_prompt_string_list = [] | ||
| for input_element, idx in zip(inputs, selected_dataset_ids): | ||
| full_prompt_string_list.append(input_element if n_shot == 0 else demonstration_string + "\n\n" + input_element) | ||
| full_prompt_string_list.append(demonstration_string + "\n\n" + input_element) |
There was a problem hiding this comment.
I think even for 0-shot, the instruction (demonstration_definition) should still be there.
|
|
||
| # 2. update the output values if needed, i.e. if the multiple choice classes now have different names | ||
| assert all(len(dataset[idx]['output']) == 1 for idx in selected_dataset_ids) | ||
| assert all(len(dataset[idx]['output']) >= 1 for idx in selected_dataset_ids) |
There was a problem hiding this comment.
Be a bit lenient here. This doesn't hold for some generation tasks. When that happens, only take the first one.
There was a problem hiding this comment.
Wouldn't it be better to raise some kind of warning in the > 1 case? Otherwise the code will silently take the first output and you might never know it happened
There was a problem hiding this comment.
Yeah makes sense. Added a warning below.
|
bertscore added |
|
Amazing, thank you for extending FormatSpread for generation tasks!!! |
2 participants
I will add bertscore later. But also you don't have to merge this since this doesn't block me. Only if you think this is helpful.