Connection refused when testing set is large

[George0828Zhang]

Hi,
I noticed that when using dev/test set with large number of sentences (e.g. CoVoST2's dev/test generally have 15k sentences), the evaluation will hang due to the following error:

2021-11-27 03:49:51 | INFO     | simuleval.scorer | Number of sentences: 15492
HTTPConnectionPool(host='localhost', port=12347): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fec33199ca0>: Failed to establish a new connection: [Errno 111] Connection refused'))

After some printing here and there, I suspect this error comes from here . however I'm not familiar with web app so I have no idea how to correct/avoid this error.

Is there any easy solution to this? Much appreciated. @xutaima

EDIT:
Forgot to mention that this does not happen if I use a subset of sentences e.g. 1500 sentences.

[EricLina]

I met the same problem , I am wondering is there a way to run in local without servers or Web

[xutaima]

Hi, I noticed that when using dev/test set with large number of sentences (e.g. CoVoST2's dev/test generally have 15k sentences), the evaluation will hang due to the following error:

2021-11-27 03:49:51 | INFO     | simuleval.scorer | Number of sentences: 15492
HTTPConnectionPool(host='localhost', port=12347): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fec33199ca0>: Failed to establish a new connection: [Errno 111] Connection refused'))

After some printing here and there, I suspect this error comes from here . however I'm not familiar with web app so I have no idea how to correct/avoid this error.

Is there any easy solution to this? Much appreciated. @xutaima

EDIT: Forgot to mention that this does not happen if I use a subset of sentences e.g. 1500 sentences.

@George0828Zhang Thanks for reporting the issue! I haven't tried a test set of this size. And the error you posted looks like the server side is down (Maybe because of the the memory issue). I will found a work around recently and let you know if there is any update.

[xutaima]

I met the same problem , I am wondering is there a way to run in local without servers or Web

Hi @1190301804 currently there is only server-client setup. Local setup is not every easy to generalize.

[EricLina]

So is there any solution except make the test size smaller ?

[Faken93]

We met the same problem, and have found the solution.

SimulEval/simuleval/cli.py

Line 187 in 2db1a59

time.sleep(3)

time.sleep(20)

The reason is that when waiting 3 seconds to start the client, the server is not initailized immediately. So the simplest way is to increase the waiting time.

As for why starting the server needs more time for SpeechAgent, the Scorer when reseting the AudioInstance would cost longer time than TextInstance.

SimulEval/simuleval/scorer/scorer.py

Line 59 in 2db1a59

self.reset()

@xutaima