Skip to content
This repository was archived by the owner on Sep 18, 2025. It is now read-only.

fix bug in computation aware latency #96

Open
zhangshaolei1998 wants to merge 1 commit into
base: main
Choose a base branch
from
Open

fix bug in computation aware latency #96

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion simuleval/evaluator/evaluator.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@
from argparse import Namespace
from pathlib import Path
from typing import Dict, Generator, Optional
from copy import deepcopy

import pandas
import yaml
Expand Down Expand Up @@ -283,7 +284,9 @@ def from_args(cls, args):
latency_scorers = {}
use_ref_len = not args.no_use_ref_len
for name in args.latency_metrics:
latency_scorers[name] = get_scorer_class("latency", name).from_args(args)
latency_args=deepcopy(args)
latency_args.computation_aware=False
latency_scorers[name] = get_scorer_class("latency", name).from_args(latency_args)
Copy link
Contributor

@xutaima xutaima Dec 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this line always setting latency_args.computation_aware as false?

Copy link
Author

@zhangshaolei1998 zhangshaolei1998 Dec 7, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, this line is for creating non-ca latency, so computation_aware should be set to False. Line 290 is used to determine whether ca latency needs to be created based on the incoming parameters (--computation-aware).

We have tested it on simultaneous text-to-text speech-to-text and speech-to-text tasks, and can partially solve the bug when using --computation-aware.

if args.computation_aware:
latency_scorers[name + "_CA"] = get_scorer_class("latency", name)(
computation_aware=True, use_ref_len=use_ref_len
Expand Down