how to perform batch inference

[vigneshmj1997]

in model.py line number 443. if we do a batch inference we will be comparing vector to a scalar resulting in an error is there any way to overcome this and perform inference in a batch.
thank you in advance.

[dambaquyen96]

I met the same problem. Has anyone been working on this?

[delgerdalai]

It might be helpful - PR #176

Issue - #177

[xinshengwang]

in model.py line number 443. if we do a batch inference we will be comparing vector to a scalar resulting in an error is there any way to overcome this and perform inference in a batch.
thank you in advance.

Hi, I met the same problem. Have you solved it?

[xxoospring]

@ShawnWXS I also met this problem, have you solved it?

[CookiePPP]

Ditto!
Anybody done batch inference yet or have any starting code?

edit: Did it, message if you need the details. model.py and the python notebook need some messy modifications.

[ksaidin]

@CookiePPP , is there any chance to look at your modifications?

I am trying to build a hosting mechanism where my server had checkpoint and waveglow loading already granted, so that I can perform batching inference which starts from text_to_sequence part.

[CookiePPP]

@ksaidin
I've only tested it with same text inputs.
I follow the tacotron forward() example and simply slice each item in the output batch by the gate_outputs max value.

e.g;

saved_postnet_outputs = []

# run tacotron with batched text_sequence input and get batched and padded outputs.
mel_batch_outputs, mel_batch_outputs_postnet, gate_batch_outputs, alignments_batch = model.inference(text_sequence)

batch = zip(
    mel_batch_outputs.split(1,dim=0),
    mel_batch_outputs_postnet.split(1,dim=0),
    gate_batch_outputs.split(1,dim=0),
    alignments_batch.split(1,dim=0),)
# split batch into individual inputs

for mel_outputs, mel_outputs_postnet, gate_outputs, alignments in batch:

    # end each clip when the speaker stops speaking
    gate_outputs[:,:10] = 0 # values in the first 10 frames are not valid
    gate_outputs[gate_outputs<0.2] = 0 # values < 0.2 are also ignored
    mel_outputs = mel_outputs[:,:,:gate_outputs.argmax()]
    mel_outputs_postnet = mel_outputs_postnet[:,:,:gate_outputs.argmax()]
    alignments = alignments[:,:gate_outputs.argmax(),:]

    saved_postnet_outputs.append(mel_outputs_postnet)

in this example, saved_postnet_outputs is an array of [1, n_mel_channels, decoder_timestep] shape tensors.

and inside model.py

gate_outputs += [gate_output.squeeze(1)]
...
if torch.sigmoid(gate_output).mean() > self.gate_threshold:
    break

I note, gate_threshold may not work for you to decide when to stop generating since I'm using same text inputs and discard any clips that fail to align to all timesteps (so if it was cut off before the end, that clip is automatically discarded before I see it).

[xinshengwang]

I made slight changes to make it can be used to perform batch inference directly: https://github.com/xinshengwang/Tacotron2_batch_inference

[rafaelvalle]

Closing due to inactivity. Thank you the batch inference version.