nnet3-rnnlm lattice rescoring draft #1906
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,89 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Copyright 2012 Johns Hopkins University (author: Daniel Povey) Tony Robinson | ||
| # 2015 Guoguo Chen | ||
| # 2017 Hainan Xu | ||
|
|
||
| # This script trains LMs on the swbd LM-training data. | ||
| # This script takes no command-line arguments but takes the --cmd option. | ||
|
|
||
| # Begin configuration section. | ||
| cmd=run.pl | ||
|
There was a problem hiding this comment. this is not right. You should be sourcing cmd.sh and using e.g. train_cmd from there. |
||
| dir=exp/rnnlm_lstm_d | ||
| embedding_dim=800 | ||
| lstm_rpd=200 | ||
| lstm_nrpd=200 | ||
| stage=-10 | ||
| train_stage=-10 | ||
|
|
||
| . utils/parse_options.sh | ||
|
|
||
| text=data/train/text | ||
| lexicon=data/local/dict_nosp/lexiconp.txt | ||
| text_dir=data/rnnlm/text_nosp | ||
| mkdir -p $dir/config | ||
| set -e | ||
|
|
||
| for f in $text $lexicon; do | ||
| [ ! -f $f ] && \ | ||
| echo "$0: expected file $f to exist; search for local/wsj_extend_dict.sh in run.sh" && exit 1 | ||
| done | ||
|
|
||
| if [ $stage -le 0 ]; then | ||
| mkdir -p $text_dir | ||
| echo -n >$text_dir/dev.txt | ||
| # hold out one in every 500 lines as dev data. | ||
| cat $text | cut -d ' ' -f2- | awk -v text_dir=$text_dir '{if(NR%500 == 0) { print >text_dir"/dev.txt"; } else {print;}}' >$text_dir/swbd.txt | ||
| fi | ||
|
|
||
| if [ $stage -le 1 ]; then | ||
| # the training scripts require that <s>, </s> and <brk> be present in a particular | ||
| # order. | ||
| awk '{print $1}' $lexicon | sort | uniq | \ | ||
| awk 'BEGIN{print "<eps> 0";print "<s> 1"; print "</s> 2"; print "<brk> 3";n=4;} {print $1, n++}' \ | ||
| >$dir/config/words.txt | ||
| # words that are not present in words.txt but are in the training or dev data, will be | ||
| # mapped to <SPOKEN_NOISE> during training. | ||
| echo "<unk>" >$dir/config/oov.txt | ||
|
|
||
| cat > $dir/config/data_weights.txt <<EOF | ||
| swbd 1 1.0 | ||
| EOF | ||
|
|
||
| rnnlm/get_unigram_probs.py --vocab-file=$dir/config/words.txt \ | ||
| --unk-word="<unk>" \ | ||
| --data-weights-file=$dir/config/data_weights.txt \ | ||
| $text_dir | awk 'NF==2' >$dir/config/unigram_probs.txt | ||
|
|
||
| # choose features | ||
| rnnlm/choose_features.py --unigram-probs=$dir/config/unigram_probs.txt \ | ||
| --use-constant-feature=true \ | ||
| --special-words='<s>,</s>,<brk>,<unk>,[noise],[laughter]' \ | ||
| $dir/config/words.txt > $dir/config/features.txt | ||
|
|
||
| cat >$dir/config/xconfig <<EOF | ||
| input dim=$embedding_dim name=input | ||
| relu-renorm-layer name=tdnn1 dim=$embedding_dim input=Append(0, IfDefined(-1)) | ||
| fast-lstmp-layer name=lstm1 cell-dim=$embedding_dim recurrent-projection-dim=$lstm_rpd non-recurrent-projection-dim=$lstm_nrpd | ||
| relu-renorm-layer name=tdnn2 dim=$embedding_dim input=Append(0, IfDefined(-3)) | ||
| fast-lstmp-layer name=lstm2 cell-dim=$embedding_dim recurrent-projection-dim=$lstm_rpd non-recurrent-projection-dim=$lstm_nrpd | ||
| relu-renorm-layer name=tdnn3 dim=$embedding_dim input=Append(0, IfDefined(-3)) | ||
| output-layer name=output include-log-softmax=false dim=$embedding_dim | ||
| EOF | ||
| rnnlm/validate_config_dir.sh $text_dir $dir/config | ||
| fi | ||
|
|
||
| if [ $stage -le 2 ]; then | ||
| # the --unigram-factor option is set larger than the default (100) | ||
| # in order to reduce the size of the sampling LM, because rnnlm-get-egs | ||
| # was taking up too much CPU (as much as 10 cores). | ||
|
There was a problem hiding this comment. can you check if this is still the case (i.e. without this is it still taking up too many cores?) There was a problem hiding this comment. Just ran this with the default 100 and it's not using more than one core. Deleting the comment. Should I change the 200 to the default 100? There was a problem hiding this comment. You can remove the option. |
||
| rnnlm/prepare_rnnlm_dir.sh --unigram-factor 200.0 \ | ||
| $text_dir $dir/config $dir | ||
| fi | ||
|
|
||
| if [ $stage -le 3 ]; then | ||
| rnnlm/train_rnnlm.sh --num-jobs-initial 1 --num-jobs-final 3 \ | ||
| --stage $train_stage --num-epochs 10 --cmd "queue.pl" $dir | ||
| fi | ||
|
|
||
| exit 0 | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,32 @@ | ||
| #!/bin/bash | ||
|
|
||
|
There was a problem hiding this comment. I'm not sure that I like the fact that the training script and rescoring script are separate. |
||
| n=50 | ||
| ngram_order=4 | ||
| rnndir= | ||
| id=rnn | ||
|
|
||
| . ./utils/parse_options.sh | ||
| . ./cmd.sh | ||
| . ./path.sh | ||
|
|
||
| set -e | ||
|
|
||
| LM=fsh_sw1_tg | ||
| rnndir=exp/rnnlm_lstm_d | ||
|
|
||
|
|
||
| for decode_set in eval2000; do | ||
| dir=exp/chain/tdnn_lstm_1e_sp | ||
| decode_dir=${dir}/decode_${decode_set}_$LM | ||
|
|
||
| # Lattice rescoring | ||
| steps/lmrescore_rnnlm_lat.sh \ | ||
| --cmd "$decode_cmd --mem 16G" \ | ||
| --rnnlm-ver kaldirnnlm --weight 0.5 --max-ngram-order $ngram_order \ | ||
| data/lang_$LM $rnndir \ | ||
| data/${decode_set}_hires ${decode_dir} \ | ||
| ${decode_dir}.nnet3rnnlm.lat.${ngram_order}gram | ||
|
|
||
| done | ||
|
|
||
| wait | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1 @@ | ||
| ../../../scripts/rnnlm/ |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,149 @@ | ||
| // latbin/lattice-lmrescore-kaldi-rnnlm.cc | ||
|
|
||
| // Copyright 2017 Johns Hopkins University (author: Daniel Povey) | ||
| // Hainan Xu | ||
|
There was a problem hiding this comment. should have date on each line, e.g. |
||
| // Yiming Wang | ||
|
|
||
| // See ../../COPYING for clarification regarding multiple authors | ||
| // | ||
| // Licensed under the Apache License, Version 2.0 (the "License"); | ||
| // you may not use this file except in compliance with the License. | ||
| // You may obtain a copy of the License at | ||
| // | ||
| // http://www.apache.org/licenses/LICENSE-2.0 | ||
| // | ||
| // THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY | ||
| // KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED | ||
| // WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE, | ||
| // MERCHANTABLITY OR NON-INFRINGEMENT. | ||
| // See the Apache 2 License for the specific language governing permissions and | ||
| // limitations under the License. | ||
|
|
||
|
|
||
| #include "base/kaldi-common.h" | ||
| #include "fstext/fstext-lib.h" | ||
| #include "lat/kaldi-lattice.h" | ||
| #include "lat/lattice-functions.h" | ||
| #include "rnnlm/rnnlm-lattice-rescoring.h" | ||
| #include "util/common-utils.h" | ||
| #include "nnet3/nnet-utils.h" | ||
|
|
||
| int main(int argc, char *argv[]) { | ||
| try { | ||
| using namespace kaldi; | ||
| typedef kaldi::int32 int32; | ||
| typedef kaldi::int64 int64; | ||
|
|
||
| const char *usage = | ||
| "Rescores lattice with rnnlm. The LM will be wrapped into the\n" | ||
|
There was a problem hiding this comment. This documentation is too code-level. Remove most of this text, and instead just mention the script in steps/ that calls it. |
||
| "DeterministicOnDemandFst interface and the rescoring is done by\n" | ||
| "composing with the wrapped LM using a special type of composition\n" | ||
| "algorithm. Determinization will be applied on the composed lattice.\n" | ||
| "\n" | ||
| "Usage: lattice-lmrescore-kaldi-rnnlm [options] <embedding-file> <rnnlm-wordlist> \\\n" | ||
| " <word-symbol-table-rxfilename> <lattice-rspecifier> \\\n" | ||
| " <raw-rnnlm-rxfilename> <lattice-wspecifier>\n" | ||
| " e.g.: lattice-lmrescore-kaldi-rnnlm --lm-scale=-1.0 word_embedding.mat \\\n" | ||
| " rnn_words.txt fst_words.txt ark:in.lats rnnlm ark:out.lats\n"; | ||
|
|
||
| ParseOptions po(usage); | ||
| int32 max_ngram_order = 3; | ||
| BaseFloat lm_scale = 1.0; | ||
|
|
||
| po.Register("lm-scale", &lm_scale, "Scaling factor for language model " | ||
| "costs; frequently 1.0 or -1.0"); | ||
| po.Register("max-ngram-order", &max_ngram_order, "If positive, limit the " | ||
| "rnnlm context to the given number, -1 means we are not going " | ||
|
There was a problem hiding this comment. this is not quite accurate, we don't limit, we identify contexts that are equal in the n most recent words. |
||
| "to limit it."); | ||
|
|
||
| po.Read(argc, argv); | ||
|
|
||
| if (po.NumArgs() != 6) { | ||
| po.PrintUsage(); | ||
| exit(1); | ||
| } | ||
|
|
||
| std::string lats_rspecifier, rnn_wordlist, word_embedding_rxfilename, | ||
| word_symbols_rxfilename, rnnlm_rxfilename, lats_wspecifier; | ||
|
|
||
| word_embedding_rxfilename = po.GetArg(1); | ||
| rnn_wordlist = po.GetArg(2); | ||
| word_symbols_rxfilename = po.GetArg(3); | ||
| lats_rspecifier = po.GetArg(4); | ||
| rnnlm_rxfilename = po.GetArg(5); | ||
| lats_wspecifier = po.GetArg(6); | ||
|
|
||
| // Reads the language model. | ||
| kaldi::nnet3::Nnet rnnlm; | ||
| ReadKaldiObject(rnnlm_rxfilename, &rnnlm); | ||
|
|
||
| if (!IsSimpleNnet(rnnlm)) | ||
| KALDI_ERR << "Input RNNLM in " << rnnlm_rxfilename | ||
| << " is not the type of neural net we were looking for; " | ||
|
There was a problem hiding this comment. Your Jedi mind tricks won't work on me. |
||
| "failed IsSimpleNnet()."; | ||
|
|
||
| CuMatrix<BaseFloat> word_embedding_mat; | ||
| ReadKaldiObject(word_embedding_rxfilename, &word_embedding_mat); | ||
|
|
||
| const nnet3::DecodableRnnlmSimpleLoopedComputationOptions opts; | ||
| const nnet3::DecodableRnnlmSimpleLoopedInfo info(opts, rnnlm, word_embedding_mat); | ||
|
|
||
| // Reads and writes as compact lattice. | ||
| SequentialCompactLatticeReader compact_lattice_reader(lats_rspecifier); | ||
| CompactLatticeWriter compact_lattice_writer(lats_wspecifier); | ||
|
|
||
| int32 n_done = 0, n_fail = 0; | ||
| for (; !compact_lattice_reader.Done(); compact_lattice_reader.Next()) { | ||
| std::string key = compact_lattice_reader.Key(); | ||
| CompactLattice clat = compact_lattice_reader.Value(); | ||
|
There was a problem hiding this comment. Due to changes in how the reader classes are implemented, you can now make 'clat' a reference, and delete the line that says FreeCurrent(). You can do the same in wherever you got this code from, as well. |
||
| compact_lattice_reader.FreeCurrent(); | ||
|
|
||
| if (lm_scale != 0.0) { | ||
| // Before composing with the LM FST, we scale the lattice weights | ||
| // by the inverse of "lm_scale". We'll later scale by "lm_scale". | ||
| // We do it this way so we can determinize and it will give the | ||
| // right effect (taking the "best path" through the LM) regardless | ||
| // of the sign of lm_scale. | ||
| fst::ScaleLattice(fst::GraphLatticeScale(1.0 / lm_scale), &clat); | ||
| ArcSort(&clat, fst::OLabelCompare<CompactLatticeArc>()); | ||
|
|
||
| // Wraps the rnnlm into FST. We re-create it for each lattice to prevent | ||
| // memory usage increasing with time. | ||
| nnet3::KaldiRnnlmDeterministicFst rnnlm_fst(max_ngram_order, | ||
| rnn_wordlist, | ||
| word_symbols_rxfilename, | ||
| info); | ||
|
|
||
| // Composes lattice with language model. | ||
| CompactLattice composed_clat; | ||
| ComposeCompactLatticeDeterministic(clat, &rnnlm_fst, &composed_clat); | ||
|
|
||
| // Determinizes the composed lattice. | ||
| Lattice composed_lat; | ||
| ConvertLattice(composed_clat, &composed_lat); | ||
| Invert(&composed_lat); | ||
| CompactLattice determinized_clat; | ||
| DeterminizeLattice(composed_lat, &determinized_clat); | ||
| fst::ScaleLattice(fst::GraphLatticeScale(lm_scale), &determinized_clat); | ||
| if (determinized_clat.Start() == fst::kNoStateId) { | ||
| KALDI_WARN << "Empty lattice for utterance " << key | ||
| << " (incompatible LM?)"; | ||
| n_fail++; | ||
| } else { | ||
| compact_lattice_writer.Write(key, determinized_clat); | ||
| n_done++; | ||
| } | ||
| } else { | ||
| // Zero scale so nothing to do. | ||
| n_done++; | ||
| compact_lattice_writer.Write(key, clat); | ||
| } | ||
| } | ||
|
|
||
| KALDI_LOG << "Done " << n_done << " lattices, failed for " << n_fail; | ||
| return (n_done != 0 ? 0 : 1); | ||
| } catch(const std::exception &e) { | ||
| std::cerr << e.what(); | ||
| return -1; | ||
| } | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you should introduce the 'tuning' directory concept and introduce soft links to the best current model. Otherwise this won't scale well.