nnet3-rnnlm lattice rescoring draft #1906
some aesthetic changes; not final yet
- Loading branch information
commit 483450dc80eb35c3e4acc9ab5c788f243d041b14
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,115 @@ | ||
| #!/bin/bash | ||
|
|
||
| # Copyright 2015 Guoguo Chen | ||
| # 2017 Hainan Xu | ||
| # Apache 2.0 | ||
|
|
||
| # This script rescores lattices with RNNLM. See also rnnlmrescore.sh which is | ||
| # an older script using n-best lists. | ||
|
|
||
| # Begin configuration section. | ||
| cmd=run.pl | ||
| skip_scoring=false | ||
| max_ngram_order=4 | ||
|
There was a problem hiding this comment. Clarify via a comment that this is for the n-gram approximation in lattice rescoring, and if feasible, refer to some kind of paper that explains this concept. |
||
| N=10 | ||
|
There was a problem hiding this comment. I assume this N variable is no longer needed. |
||
| inv_acwt=12 | ||
|
There was a problem hiding this comment. we normally make this default to 10. making the default 12 would be confusing. |
||
| weight=1.0 # Interpolation weight for RNNLM. | ||
| # End configuration section. | ||
| rnnlm_ver= | ||
| #layer_string= | ||
|
|
||
| echo "$0 $@" # Print the command line for logging | ||
|
|
||
| . ./utils/parse_options.sh | ||
|
|
||
| if [ $# != 5 ]; then | ||
| echo "Does language model rescoring of lattices (remove old LM, add new LM)" | ||
| echo "with RNNLM." | ||
| echo "" | ||
| echo "Usage: $0 [options] <old-lang-dir> <rnnlm-dir> \\" | ||
| echo " <data-dir> <input-decode-dir> <output-decode-dir>" | ||
| echo " e.g.: $0 ./rnnlm data/lang_tg data/test \\" | ||
| echo " exp/tri3/test_tg exp/tri3/test_rnnlm" | ||
| echo "options: [--cmd (run.pl|queue.pl [queue opts])]" | ||
| exit 1; | ||
| fi | ||
|
|
||
| [ -f path.sh ] && . ./path.sh; | ||
|
|
||
| oldlang=$1 | ||
| rnnlm_dir=$2 | ||
| data=$3 | ||
| indir=$4 | ||
| outdir=$5 | ||
|
|
||
| rescoring_binary=lattice-lmrescore-rnnlm | ||
|
|
||
|
|
||
| if [ "$rnnlm_ver" == "kaldirnnlm" ]; then | ||
| rescoring_binary="lattice-lmrescore-kaldi-rnnlm" | ||
| first_arg="\"rnnlm-get-word-embedding $rnnlm_dir/word_feats.txt $rnnlm_dir/feat_embedding.final.mat -|\" $rnnlm_dir/config/words.txt " | ||
| fi | ||
|
|
||
| oldlm=$oldlang/G.fst | ||
| if [ -f $oldlang/G.carpa ]; then | ||
| oldlm=$oldlang/G.carpa | ||
| elif [ ! -f $oldlm ]; then | ||
| echo "$0: expecting either $oldlang/G.fst or $oldlang/G.carpa to exist" &&\ | ||
| exit 1; | ||
| fi | ||
|
|
||
| [ ! -f $oldlm ] && echo "$0: Missing file $oldlm" && exit 1; | ||
| [ ! -f $rnnlm_dir/final.raw ] && echo "$0: Missing file $rnnlm_dir/final.raw" && exit 1; | ||
| [ ! -f $rnnlm_dir/feat_embedding.final.mat ] && [ ! -f $rnnlm_dir/word_embedding.final.mat ] && echo "$0: Missing word embedding file" && exit 1; | ||
|
|
||
| [ ! -f $oldlang/words.txt ] &&\ | ||
| echo "$0: Missing file $oldlang/words.txt" && exit 1; | ||
| ! ls $indir/lat.*.gz >/dev/null &&\ | ||
| echo "$0: No lattices input directory $indir" && exit 1; | ||
| awk -v n=$0 -v w=$weight 'BEGIN {if (w < 0 || w > 1) { | ||
| print n": Interpolation weight should be in the range of [0, 1]"; exit 1;}}' \ | ||
| || exit 1; | ||
|
|
||
| oldlm_command="fstproject --project_output=true $oldlm |" | ||
|
There was a problem hiding this comment. I believe this projection is no longer necessary because IIRC the code does the projection internally. There was a problem hiding this comment. Hmm, this is not the pruned version of lattice-rescoring and it uses the old latbin binaries. |
||
|
|
||
| acwt=`perl -e "print (1.0/$inv_acwt);"` | ||
|
|
||
| word_embedding= | ||
| if [ -f $rnnlm_dir/word_embedding.final.mat ]; then | ||
| word_embedding=$rnnlm_dir/word_embedding.final.mat | ||
| else | ||
| word_embedding="\"rnnlm-get-word-embedding $rnnlm_dir/word_feats.txt $rnnlm_dir/feat_embedding.final.mat -|\"" | ||
|
There was a problem hiding this comment. To avoid the need for escaping, I suggest using single quotes for the inner quotes. |
||
| fi | ||
|
|
||
| mkdir -p $outdir/log | ||
| nj=`cat $indir/num_jobs` || exit 1; | ||
| cp $indir/num_jobs $outdir | ||
|
|
||
| oldlm_weight=`perl -e "print -1.0 * $weight;"` | ||
|
There was a problem hiding this comment. I generally prefer |
||
| if [ "$oldlm" == "$oldlang/G.fst" ]; then | ||
| $cmd JOB=1:$nj $outdir/log/rescorelm.JOB.log \ | ||
| lattice-lmrescore --lm-scale=$oldlm_weight \ | ||
| "ark:gunzip -c $indir/lat.JOB.gz|" "$oldlm_command" ark:- \| \ | ||
| $rescoring_binary --lm-scale=$weight \ | ||
| --max-ngram-order=$max_ngram_order \ | ||
| $oldlang/words.txt ark:- $rnnlm_dir/config/words.txt $word_embedding "$rnnlm_dir/final.raw" \ | ||
| "ark,t:|gzip -c>$outdir/lat.JOB.gz" || exit 1; | ||
| else | ||
| $cmd JOB=1:$nj $outdir/log/rescorelm.JOB.log \ | ||
| lattice-lmrescore-const-arpa --lm-scale=$oldlm_weight \ | ||
| "ark:gunzip -c $indir/lat.JOB.gz|" "$oldlm" ark:- \| \ | ||
| $rescoring_binary --lm-scale=$weight \ | ||
| --max-ngram-order=$max_ngram_order \ | ||
| $oldlang/words.txt ark:- $rnnlm_dir/config/words.txt $word_embedding "$rnnlm_dir/final.raw" \ | ||
| "ark,t:|gzip -c>$outdir/lat.JOB.gz" || exit 1; | ||
| fi | ||
| if ! $skip_scoring ; then | ||
| err_msg="Not scoring because local/score.sh does not exist or not executable." | ||
| [ ! -x local/score.sh ] && echo $err_msg && exit 1; | ||
| local/score.sh --cmd "$cmd" $data $oldlang $outdir | ||
| else | ||
| echo "Not scoring because requested so..." | ||
| fi | ||
|
|
||
| exit 0; | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't like the 'rnnlm' in the name here because it's redundant-- it's already in a directory called 'rnnlm'. You could call it 'lmrescore_lat.sh', but IMO the name 'lmrescore.sh' would be more ideal, and you could then have the n-best one be called 'lmrescore_nbest.sh', if needed.