Newnnet3 #1

mrsrikanth · 2019-09-15T13:31:00Z

No description provided.

danpovey · 2019-10-05T12:32:58Z

OK, here are my initial thoughts.
I don't know whether some of those decisions I'm complaining about were ones I originally made... possibly. Sorry for taking so long with this!

mrsrikanth · 2019-10-06T16:00:08Z

Hi @danpovey where are your comments? Or was it just the above one?

danpovey

sorry, missed the actual review

danpovey · 2019-10-05T08:11:55Z

egs/mini_librispeech/s5b/cmd.sh

+# or search for the string 'default_config' in utils/queue.pl or utils/slurm.pl.
+
+export train_cmd="queue.pl -l q1d -l io_big -V"
+export decode_cmd="queue.pl -l q1d -l io_big -V"


I'd prefer if you don't put things here that are specific to your grid.
Also I'm not convinced that you need to copy the whole mini_librispeech recipe if you are just changing the chain stuff. We'll have this issue elsewhere too. Perhaps we could use the subdirectory name chain2 for the new script style.

I have now deleted the s5b recipe. I have create steps/nnet3/chain2 and a corresponding soft link steps/chain2. I have also created local/chain2 and moved the new code to all these folders.

danpovey · 2019-10-05T08:15:37Z

egs/mini_librispeech/s5b/local/chain/tuning/run_cnn_tdnn_1a.sh

+  # matched topology (since it gets the topology file from the model).
+  utils/mkgraph.sh \
+    --self-loop-scale 1.0 data/lang_test_tgsmall \
+    $tree_dir $tree_dir/graph_tgsmall || exit 1;


this file looks unchanged to me, vs. the old one... makes it hard to see what the changes are if lots of files are duplicated. (And we wouldn't want to duplicate unnecessary files in the real thing.) But I'll try to figure out where the important stuff is.

In the updated PR, you will no longer have 100+ files to review. Sorry about this.

danpovey · 2019-10-05T08:24:26Z

egs/wsj/s5/steps/nnet3/chain/train2.sh

+#!/bin/bash
+
+# Copyright   2019  Johns Hopkins University (Author: Daniel Povey).  Apache 2.0.
+


Re the naming of this: perhaps it would be better to have steps/nnet3/chain2/train.sh, and put the other things related to this in the chain2 directory. easier for discoverability and gives more structure.

As mentioned above, I created steps/nnet3/chain2 in egs/wsj/s5. In both minilibrespeech and swb, there are now local/chain2 folders.

danpovey · 2019-10-05T08:28:07Z

egs/swbd/s5c/local/chain/tuning/run_tdnn_8k.sh

@@ -0,0 +1,385 @@
+#!/bin/bash
+
+# run_tdnn_8k.sh is like run_tdnn_7k.sh but uses new kaldi recipe.


I don't think you have understood the naming scheme right... we normally go 1a, 1b etc., and if we use up the letters, go to 2a. We don't normally increase the number while leaving the letter the same.
Let's take "chaina" out of all of the names, but we can use chain. For example: exp/chain2. And steps/chain2 -> steps/nnet3/chain2, as a soft link, just like steps/chain -> steps/nnet3/chain.

The problem was that I was trying to re-use the local/chain/tuning folders. But now, I have changed the folder organisation, so the numbering will be consistent.

danpovey · 2019-10-05T08:30:12Z

egs/swbd/s5d/RESULTS

@@ -0,0 +1,54 @@
+# Baseline results with tdnn_7k recipe
+  cat exp/chain/tdnn_7k_sp/decode_eval2000_sw1_fsh_fg/score_*/*swbd.filt.sys | fgrep Sum | sort -k11,11 -g | head


This is not how we display the results... comparing across all acwt's is very unreadable. Instead you can use local/chain/compare_wer.sh, or use the commands in the old RESULTS file ,,, things like
grep Sum exp/chain/tdnn_7k_sp/decode_eval2000_sw1_fsh_fg/score_*/*swbd.filt.sys | utils/best_wer.sh

This is fixed now. But there will no longer be a new recipe for swbd. I just modified the RESULTS file in swbd/s5c

danpovey · 2019-10-05T08:33:30Z

egs/swbd/s5d/cmd.sh

@@ -0,0 +1,17 @@
+# you can change cmd.sh depending on what type of queue you are using.
+# If you have no queueing system and want to run on a local machine, you
+# can change all instances 'queue.pl' to run.pl (but be careful and run


Please don't duplicate the entire swbd recipe (c->d) unless there is something going on early in data preparation that would make things incompatible. We need to structure these new scripts so that they can slot into existing recipes that have already been run and that already have scripts, without invalidating the existing stuff.

I removed swbd/s5d recipes. There no new recipes in the updated PR.

danpovey · 2019-10-05T08:34:22Z

egs/wsj/s5/steps/diagnostic/analyze_lats.sh

 #end configuration section.

 echo "$0 $@"  # Print the command line for logging
+exit 0


If there was an error in this script LMK what it was!
I think we have fixed bugs in this script in the past... possibly your code base was not merged with the latest master changes when you ran this stuff.

I have reverted this. This was a problem for me at Idiap, but I verified with other people here. They don't have this problem. So, I'll create an issue in the main Kaldi repo if I get this error again.

danpovey · 2019-10-05T08:34:58Z

egs/wsj/s5/steps/diagnostic/analyze_alignments.sh

 echo "$0 $@"  # Print the command line for logging

-[ -f ./path.sh ] && . ./path.sh
+#[ -f ./path.sh ] && . ./path.sh


you might want to revert these changes.
LMK what the problem is.. we should fix it in the right way.

This was a constant problem for me when running things on the grid here at Idiap. So, I commented it. Sourcing the path multiple times sends the jobs to Eqw. I have reverted this in the PR to master. Again, I will create an issue in Kaldi if I can reproduce the problem consistently.

danpovey · 2019-10-05T08:41:37Z

egs/wsj/s5/steps/nnet3/chain/compute_preconditioning_matrix.sh

+        fi
+        echo "$0: Accumulating LDA stats"
+        $cmd JOB=1:$nj $ldafolder/log/acc.JOB.log \
+                nnet3-chain-acc-lda-stats $lda_acc_opts --rand-prune=${rand_prune} \


Just an FYI: I am trying to move away from this preconditioning stuff and instead use deltas, which are easier logistically. E.g. see egs/mini_librispeech/s5/local/chain/tuning/run_tdnn_1j.sh.

Yes, you have mentioned this to me over e-mail. But I had to implement this to make sure there is no difference from the existing scripts, so that I can make a fair comparison. I can remove it if you want.

danpovey · 2019-10-05T08:43:01Z

egs/wsj/s5/steps/nnet3/chain/data_prep_common.sh

+set -euo pipefail
+
+stage=0
+train_set=train_clean_5


This doesn't look like it belongs in steps! It looks like a local script! Also I would like to move away from speed perturbation and toward noise and reverb augmentation, so I'm not sure that I want to bake the speed perturbation into the script like this.

Yes, this was a mistake. I didn't use this script anywhere (at least not the one in this location). I have removed it from the repo.

mrsrikanth · 2019-10-07T15:47:31Z

Hi Dan,
Thanks for all the comments. It will take me a couple of days to look at them and reply as I'm facing a couple of deadlines today and tomorrow.

Srikanth

danpovey · 2019-10-08T01:46:12Z

Thanks!

…

On Mon, Oct 7, 2019 at 8:47 AM Srikanth M R ***@***.***> wrote: Hi Dan, Thanks for all the comments. It will take me a couple of days to look at them and reply as I'm facing a couple of deadlines today and tomorrow. Srikanth — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1?email_source=notifications&email_token=AAZFLO7DPENXKPSWWG34OALQNNKZJA5CNFSM4IW2TVEKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAQ23TQ#issuecomment-539078094>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZFLO7PGZXWFIIAMCT6VHTQNNKZJANCNFSM4IW2TVEA> .

… egs/mini_librispeech

… build_tree.sh from run_tdnn_2a.sh

align_lats.sh get_raw_egs.sh process_egs.sh randomize_egs.sh train2.sh (didn't know what else to call it) validate_processed_egs.sh validate_randomized_egs.sh validate_raw_egs.sh choose_egs_to_merge.py get_train_schedule.py

…related variables

…ain-get-egs

…ptions

…ibrespeech recipe

local/chain/tuning/run_tdnn_2a.sh: works now, but the WER is 5% worse than expected local/chain/tuning/run_tdnn_1a_sans_ivectors.sh: version of local/chain/tuning/run_tdnn_1a.sh that doesn't use ivectors local/chain/data_prep_common.sh: useful for new type of chain scripts

…raining script

…te LDA for chain models

…for chain model scripts

…tors. looks for transition model in two places.

…f commented code. but committing it because it works and the code is moving towards what we want

…d to chain

…es to chain

…hain

…_input_frames is now obtained from data folder

…ne2.cc the latter is useful for final model combination in the new chain training recipes the former is a header file I forgot to add under git

…eps/nnet3/chain2/

…5b/local/.., but there is no need to have a new recipe

… scripts

…nn_7k.sh

…steps

- deleted s5b recipe in minlibrespeech

… is now the same as what is there in master

…h_dir from options

…/chain2/tuning/run_tdnn_7k.sh

…s to be reviewed for PR). these files originally had changes adapted to the Idiap environment: egs/wsj/s5/steps/nnet3/chain/get_egs.sh egs/wsj/s5/steps/nnet3/chain/train.py egs/wsj/s5/steps/online/nnet2/train_ivector_extractor.sh egs/swbd/s5c/local/chain/tuning/run_tdnn_7k.sh

egs/wsj/s5/steps/online/nnet2/train_diag_ubm.sh egs/wsj/s5/steps/online/nnet2/train_ivector_extractor.sh this simplifies PR a bit

… from the commits, it was there in the chaina branch, but it is no longer useful

mrsrikanth requested a review from danpovey September 15, 2019 13:31

danpovey reviewed Oct 6, 2019

View reviewed changes

Srikanth MADIKERI added 24 commits November 4, 2019 17:28

replicating mini_librispeech/s5 to mini_librispeech/s5b

75dcf22

adding run_tdnn_2a.sh, data_pre_common.sh and get_model_context.sh to…

d3087a8

… egs/mini_librispeech

egs/mini_librispeech/s5b: removing num-clusters options while calling…

54a3097

… build_tree.sh from run_tdnn_2a.sh

adding many new source files from PR:

b747ba9

align_lats.sh get_raw_egs.sh process_egs.sh randomize_egs.sh train2.sh (didn't know what else to call it) validate_processed_egs.sh validate_randomized_egs.sh validate_raw_egs.sh choose_egs_to_merge.py get_train_schedule.py

train2.sh: chaning chaina references to chain

64c6cc5

egs/wsj5/steps/nnet3/chain/get_model_context.sh: removed bottom nnet …

170eee3

…related variables

steps/nnet3/chain/get_raw_egs.sh: removed long-key option to nnet3-ch…

5b90e80

…ain-get-egs

steps/nnet3/chain/get_raw_egs.sh: added cmvn and online_ivector_dir o…

32de786

…ptions

egs/wsj5/steps/nnet3/chain/get_raw_egs.sh: working version with minil…

0e32c4f

…ibrespeech recipe

modifications to process_egs.sh

7d27b06

steps/nnet3/chain/train2.sh: working version of the new chain model t…

8ebe81a

…raining script

egs/wsj/s5/steps/nnet3/chain/compute_preconditioning_matrix.sh: compu…

1f9ceb4

…te LDA for chain models

steps/libs/nnet3/train/dropout_schedule.py: compute dropout schedule …

b6717cf

…for chain model scripts

steps/nnet3/chain/get_raw_egs.sh: added optoins for cmvn, online-ivec…

71c7f14

…tors. looks for transition model in two places.

steps/nnet3/chain/internal/choose_egs_to_merge.py: now an executable

2831152

steps/nnet3/chain/internal/get_train_schedule.py: now an executable

70929d8

steps/nnet3/chain/process_egs.sh: an intermediate version with lots o…

1a9c3a5

…f commented code. but committing it because it works and the code is moving towards what we want

steps/nnet3/chain/process_egs.sh: changed chaina references to chain

10d83ab

steps/nnet3/chain/validate_processed_egs.sh: chaina references change…

6f60a35

…d to chain

steps/nnet3/chain/validate_randomized_egs.sh: changed chaina referenc…

4967bfe

…es to chain

steps/nnet3/chain/validate_raw_egs.sh: changed chaina references to c…

fe0594d

…hain

steps/nnet3/chain/get_raw_egs.sh: working version of get_raw_egs. num…

9120a8b

…_input_frames is now obtained from data folder

steps/nnet3/chain/process_egs.sh: working version of process_egs.sh

1dc4813

Srikanth MADIKERI added 30 commits December 10, 2019 04:38

adding nnet3/nnet-chain-diagnostics2.h and chainbin/nnet3-chain-combi…

d41c359

…ne2.cc the latter is useful for final model combination in the new chain training recipes the former is a header file I forgot to add under git

moved code for the new training scripts from steps/nnet3/chain/ to st…

0bec666

…eps/nnet3/chain2/

added soft link to steps/nnet3/chain2 in steps/

7bd205f

modified paths to other scripts in steps/nnet3/chain2/train2.sh

8f6546a

renamed steps/nnet3/chain2/train2.sh to steps/nnet3/chain2/train.sh

6095f63

added new script to local/chain2/tuning. earlier this recipe was in s…

29f90b9

…5b/local/.., but there is no need to have a new recipe

added softlink to best tdnn recipe for minilibrespeech with new kaldi…

163056a

… scripts

moved local/chain/tuning/run_tdnn_8k.sh to local/chain2/tuning/run_td…

1e2c129

…nn_7k.sh

reverting back steps/diagnostic/analyze_alignments.sh

0102ca8

reverting analyze_lats

9e55ef0

removing steps/nnet3/chain/data_prep_common.sh. it doesn't belong in …

e54a528

…steps

removing s5d

15690eb

RESULTS for swbd added

4ba588a

minilibrespeech RESULTS integrated into s5 recipe

1ccb3db

- added data_prep_common.sh to local/chain2

5ce849d

- deleted s5b recipe in minlibrespeech

removing the configuration section in local/chain/tuning/run_tdnn_7k.sh

6284695

removed safe-checking graphs in local/chain/tuning/run_tdnn_7k.sh. It…

55c06f3

… is now the same as what is there in master

setting graph_dir value (reverting the code in parts)

d7bf796

one more rollback to local/chain/tuning/run_tdnn_7k.sh: removing grap…

c6ede69

…h_dir from options

modifying local/chain2/tuning/run_tdnn_7k.sh to be identical to local…

2c2420b

…/chain2/tuning/run_tdnn_7k.sh

reverting 2 more files to match master:

ff4d364

egs/wsj/s5/steps/online/nnet2/train_diag_ubm.sh egs/wsj/s5/steps/online/nnet2/train_ivector_extractor.sh this simplifies PR a bit

removing egs/wsj/s5/steps/nnet3/chain/internal/choose_egs_to_merge.py…

cfda607

… from the commits, it was there in the chaina branch, but it is no longer useful

added copyright headers to code modified in mini_librespeech recipe

5989138

added copyright header to recipe in swbd

bad7351

copyright added to source code in nnet3 and chainbin

36c4883

added run_tdnn.sh for chain2 in egs/babel

ad9fb10

maded egs/babel/s5d/local/chain2/run_tdnn.sh an executable

87fe03b

tdnn recipe for babel added

e0dbf0e

merging changes after adding copyright headers

7961ea1

		#!/bin/bash

		# Copyright 2019 Johns Hopkins University (Author: Daniel Povey). Apache 2.0.

		@@ -0,0 +1,385 @@
		#!/bin/bash

		# run_tdnn_8k.sh is like run_tdnn_7k.sh but uses new kaldi recipe.

		@@ -0,0 +1,54 @@
		# Baseline results with tdnn_7k recipe
		cat exp/chain/tdnn_7k_sp/decode_eval2000_sw1_fsh_fg/score_/swbd.filt.sys \| fgrep Sum \| sort -k11,11 -g \| head

Newnnet3 #1

Are you sure you want to change the base?

Newnnet3 #1

Uh oh!

Conversation

mrsrikanth commented Sep 15, 2019

Uh oh!

danpovey commented Oct 5, 2019

Uh oh!

mrsrikanth commented Oct 6, 2019

Uh oh!

danpovey left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrsrikanth commented Oct 7, 2019

Uh oh!

danpovey commented Oct 8, 2019 via email

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants