Added Dropout component in netted to be used ephemeral connection. #1032

pegahgh · 2016-09-17T01:10:23Z

No description provided.

danpovey · 2016-09-17T01:50:37Z

src/nnet3/nnet-simple-component.h

+ public:
+  void Init(int32 dim, BaseFloat dropout_proportion = 0.0);
+
+  DropoutComponent(int32 dim, BaseFloat dp = 0.0) { Init(dim, dp); }


dp is a too-short name.

danpovey · 2016-09-17T01:50:46Z

src/nnet3/nnet-simple-component.h

+
+  DropoutComponent(int32 dim, BaseFloat dp = 0.0) { Init(dim, dp); }
+
+  DropoutComponent():dim_(0), dropout_proportion_(0.0) { }


space after colon

danpovey · 2016-09-17T02:06:00Z

You haven't added DropoutComponent to the components used in testing. So it's not being tested. It's not clear that the tests would even work without modification- it might be necessary to reset the random number generators for some of the comparisons.

danpovey · 2016-09-17T02:16:08Z

Also we need a way to set the dropout-proportions via the command line.
You could add a function
SetDropoutProportion()
in nnet-nnet.h, and add an option --set-dropout-proportion to nnet3-copy.

On Fri, Sep 16, 2016 at 6:10 PM, pegahgh notifications@github.com wrote:

You can view, comment on, or merge this pull request online at:

#1032
Commit Summary

added DropoutComponent to nnet3 for ephemeral connection test.

added DropoutComponent to nnet3 for ephemeral connection test.

fixed small issues.

fixed small issues.

fixed small issues.

fixed small issues.

File Changes

M src/nnet3/nnet-component-itf.cc
https://github.com/kaldi-asr/kaldi/pull/1032/files#diff-0 (2)

M src/nnet3/nnet-component-itf.h
https://github.com/kaldi-asr/kaldi/pull/1032/files#diff-1 (10)

M src/nnet3/nnet-nnet.cc
https://github.com/kaldi-asr/kaldi/pull/1032/files#diff-2 (11)

M src/nnet3/nnet-nnet.h
https://github.com/kaldi-asr/kaldi/pull/1032/files#diff-3 (6)

M src/nnet3/nnet-simple-component.cc
https://github.com/kaldi-asr/kaldi/pull/1032/files#diff-4 (79)

M src/nnet3/nnet-simple-component.h
https://github.com/kaldi-asr/kaldi/pull/1032/files#diff-5 (50)

M src/nnet3/nnet-test-utils.cc
https://github.com/kaldi-asr/kaldi/pull/1032/files#diff-6 (2)

Patch Links:

https://github.com/kaldi-asr/kaldi/pull/1032.patch

https://github.com/kaldi-asr/kaldi/pull/1032.diff

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#1032, or mute the thread
https://github.com/notifications/unsubscribe-auth/ADJVu0b-Jcx6xMcRhC52NcNWzi7AiAlAks5qqz4CgaJpZM4J_d_R
.

pegahgh · 2016-09-17T22:09:27Z

I added Dropout to the components in the testing
I reset random generator.
I added the --set-dropout-proportion via command line in nnet3-copy
I added SetDropoutProportion in nnet-utils.*
Done all changes. I added SetDropoutProportion, RemoveDropout and modified test to reset seed for dropout component test.
Added RemoveDropout in nnet-nnet.h

danpovey · 2016-09-18T19:12:59Z

Please push your changes, and also rebase to clean up the commit history- I prefer just one commit.

pegahgh · 2016-09-18T19:17:07Z

I pushed the new changes after rebasing (Just single commit!!!!)

On Sep 18, 2016 3:13 PM, "Daniel Povey" notifications@github.com wrote:

Please push your changes, and also rebase to clean up the commit history-
I prefer just one commit.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
#1032 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AI-PtM4Tu185pRezQbwhK__Q9DQr2Lczks5qrY1DgaJpZM4J_d_R
.

danpovey · 2016-09-20T23:18:04Z

@pegahgh, please go through all comments on this page and make sure you have addressed them-- same for the other PR you have. There are outstanding comments where the issue is not fixed.

pegahgh · 2016-09-21T00:26:52Z

@danpovey I reviewed all comments on this page and I found all the issues fixed. Maybe the issue is that I did rebase and the fixed version of codes don't appear in conversation!
Did you check the files changed directly or you just checked the conversation?

danpovey · 2016-09-21T00:30:29Z

Oh-- I just checked the conversation, it seems the files were updated if I click on the right tab.
I think it used to be the case that github would say "danpovey commented on an outdated commit" if the lines of code that I commented on had changed, but it didn't say that.

pegahgh · 2016-09-21T00:33:25Z

Maybe the reason is that I commented on conversation after committing changes, so it is not out-dated!!

danpovey · 2016-09-21T00:33:06Z

src/nnet3/nnet-nnet.cc

+  }
+}
+
+void Nnet::RemoveDropout() {


this function would not work.. nnet3 isn't just a linear sequence of components. Just remove this function for now.

danpovey · 2016-09-21T00:33:41Z

src/nnet3/nnet-component-itf.h

+ public:
+  // This function is required in testing code and in other places we need
+  // consistency in the random number generation (e.g. when optimizing
+  // validation-set performance), but check where else we call sRand().  You'll


should be srand not sRand. this appears elsewhere too, I think.

pegahgh · 2016-09-21T00:38:05Z

@danpovey I can easily test ephemeral connection using dropout on TDNN!
Probably it would be better idea, if Yiming check this connection on LSTM, since he is more expert in that area!

danpovey · 2016-09-21T00:46:51Z

Before that can be done, I need to do some coding to make it possible to remove the dropout components. I want to make this part of a more general framework for editing nnets, that will also be used in your multilingual stuff. So I think it might be better to wait a few days. But I guess if you want to start work on the scripting components, and you're not otherwise busy, it's OK with me.

danpovey · 2016-09-25T02:22:55Z

src/nnet3/nnet-nnet.h

+  // effective.
+
+  // Removes all dropout components from network, which can be used during test time.
+  void RemoveDropout();


Can you please take out this RemoveDropout() function?
I have been thinking about it, and I think the best way to do this would be at the config level: a file "remove-ephemeral-connections.config" would be created that modifies the nodes that sum over the real and emphemeral connections by replacing expressions like Add(real_node, ephmeral_node) with real_node.
Then we can clean up the model by taking out orphan nodes and components; I have written the code for this and will commit it at some point in future.

pegahgh · 2016-09-25T17:55:40Z

Removed RemoveDropout function.

danpovey · 2016-09-25T18:45:18Z

src/nnet3/nnet-simple-component.cc

+
+  KALDI_ASSERT(in_value.NumRows() == out_deriv.NumRows() &&
+               in_value.NumCols() == out_deriv.NumCols());
+  in_deriv->AddMatMatDivMat(out_deriv, out_value, in_value);


since this is adding, you should include kBackpropAdds in the Properties().
You should also add the kLinearInInput flag.

danpovey · 2016-09-25T22:10:29Z

src/nnet3/nnet-simple-component.cc

@@ -143,7 +143,9 @@ void DropoutComponent::Backprop(const std::string &debug_info,

  KALDI_ASSERT(in_value.NumRows() == out_deriv.NumRows() &&
               in_value.NumCols() == out_deriv.NumCols());
-  in_deriv->AddMatMatDivMat(out_deriv, out_value, in_value);
+  CuMatrix<BaseFloat> in_deriv_inplace(in_deriv->NumRows(), in_deriv->NumCols());


I'm not sure why you made these changes? This shouldn't have been necessary. Let me know if there was a test failure when you added the flag, the cause might be something else.

I added kBackpropAdds and kLinearInInput to Properties().
It seems AddMatMatDivMat method doesn't add this to its output, which is against what we expect in most of AddMat functions.
I didn't modify this function and I supported kBackpropAdds in backprop by having temporary matrix and adding it to in_deriv!

we may need to modify AddMatMatDivMat method to do *this += a * b / c !!

danpovey · 2016-09-25T22:20:56Z

OK, I see the problem.
The issue is that the name of the function AddMatMatDivMat is out of sync with what the function actually does: it doesn't add to the matrix, it sets it. It should be renamed to SetMatMatDivMat.
And you can remove the flag kBackpropAdds, add the flag kBackpropInPlace, and revert the code changes in the Backprop function.

pegahgh · 2016-09-26T00:17:03Z

Should I rename the function in this PR, or you plan to do this later?

danpovey · 2016-09-26T02:32:46Z

rename it

On Sun, Sep 25, 2016 at 8:17 PM, pegahgh notifications@github.com wrote:

Should I rename the function in this PR, or you plan to do this later?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1032 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ADJVu7YYK5p26PgWOWJWN4Y-hbmvykZpks5qtw8CgaJpZM4J_d_R
.

danpovey · 2016-09-27T18:27:25Z

src/cudamatrix/cu-matrix.cc

@@ -989,7 +989,7 @@ void CuMatrixBase<Real>::AddMatBlocks(Real alpha, const CuMatrixBase<Real> &A,
 /// dst = a * b / c (by element; when c = 0, dst = a)
 /// dst can be an alias of a, b or c safely and get expected result.
 template<typename Real>
-void CuMatrixBase<Real>::AddMatMatDivMat(const CuMatrixBase<Real> &A,
+void CuMatrixBase<Real>::SetMatMatDivMat(const CuMatrixBase<Real> &A,


If we don't do this now we'll forget about it-- can you also please change the kernel names to remove the add and make it set ?

vijayaditya · 2016-09-28T03:50:45Z

src/nnet3/nnet-simple-component.cc

+  return stream.str();
+}
+
+void DropoutComponent::Propagate(const ComponentPrecomputedIndexes *indexes,


In this implementation of drop-out you are using a different mask for each sample in the minibatch. I don't know if other versions of drop-out do this (can anyone clarify ?). If they don't you should probably highlight this in the comments. I think this version is better.

Hi Vijay, I am not aware of a paper where would be mentioned that the same dropout mask should be used for all the samples within the mini-batch. Yes, I am assuming that all the samples are independent, having different topologies of NN, so each sample has a different mask. Has anybody tried having it the other way? Thanks!
K.

danpovey · 2016-09-28T04:05:09Z

Karel's did. I expect they probably normally do it that way but don't know
for sure.

On Tue, Sep 27, 2016 at 11:50 PM, Vijayaditya Peddinti <
notifications@github.com> wrote:

@vijayaditya commented on this pull request.

In src/nnet3/nnet-simple-component.cc
#1032 (review):
cfl->GetValue("dropout-proportion", &dropout_proportion);

if (!ok || cfl->HasUnusedValues() || dim <= 0 ||
 dropout_proportion < 0.0 || dropout_proportion > 1.0)
KALDI_ERR << "Invalid initializer for layer of type "
         << Type() << ": \"" << cfl->WholeLine() << "\"";
Init(dim, dropout_proportion);
+}
+std::string DropoutComponent::Info() const {
std::ostringstream stream;

stream << Type() << ", dim = " << dim_
    << ", dropout-proportion = " << dropout_proportion_;
return stream.str();
+}
+void DropoutComponent::Propagate(const ComponentPrecomputedIndexes *indexes,
In this implementation of drop-out you are using a different mask for each
sample in the minibatch. I don't know if other versions of drop-out do this
(can anyone clarify ?). If they don't you should probably highlight this in
the comments. I think this version is better.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1032 (review),
or mute the thread
https://github.com/notifications/unsubscribe-auth/ADJVu9RSxmj9aSOw-32nEOTA0iyjzSc0ks5queQfgaJpZM4J_d_R
.

vijayaditya · 2016-09-30T19:31:00Z

@pegahgh I was curious about the expected completion date for this component.

danpovey · 2016-09-30T19:47:13Z

The only thing that has to be done is to rename some kernels.
(Also, to resolve some conflicts).

pegahgh · 2016-09-30T19:50:46Z

oh! I will rename them right now!

pegahgh · 2016-09-30T22:11:22Z

Done!

danpovey · 2016-09-30T22:15:11Z

src/nnet3/nnet-nnet.h

+  // random components.  You must also set srand() for this to be
+  // effective.
+
+>>>>>>> 05d2e30... added DropoutComponent to nnet3 for ephemeral connection.


looks like you did not completely resolve these conflicts!

oh sorry! fixed!

danpovey · 2016-09-30T22:44:30Z

src/nnet3/nnet-test-utils.cc

@@ -1219,6 +1219,15 @@ static void GenerateRandomComponentConfig(std::string *component_type,
        os << " self-repair-target=" << RandUniform();
      break;
    }
+<<<<<<< HEAD
+=======


and here.. check for === via grep et.

danpovey · 2016-10-01T19:48:59Z

Also, please make sure this compiles with GPU... if you had compiled, you wouldn't have missed these errors.

pegahgh · 2016-10-03T16:04:01Z

fixed type and it compiled with GPU!

danpovey reviewed Sep 17, 2016

View reviewed changes

pegahgh force-pushed the ladnn-droput branch 2 times, most recently from 7e503a2 to b42f818 Compare September 19, 2016 00:19

danpovey requested changes Sep 21, 2016

View reviewed changes

danpovey reviewed Sep 25, 2016

View reviewed changes

pegahgh force-pushed the ladnn-droput branch 3 times, most recently from 0d37b1a to e866023 Compare September 25, 2016 17:54

danpovey reviewed Sep 25, 2016

View reviewed changes

pegahgh force-pushed the ladnn-droput branch from fea7c32 to 05d2e30 Compare September 26, 2016 00:27

danpovey reviewed Sep 27, 2016

View reviewed changes

vijayaditya reviewed Sep 28, 2016

View reviewed changes

pegahgh force-pushed the ladnn-droput branch from 7bfa02e to f5691bb Compare September 30, 2016 22:08

danpovey reviewed Sep 30, 2016

View reviewed changes

pegahgh force-pushed the ladnn-droput branch from f5691bb to a48d6b3 Compare September 30, 2016 22:42

danpovey reviewed Sep 30, 2016

View reviewed changes

Added DropoutComponent in nnet3

5afe883

pegahgh force-pushed the ladnn-droput branch from a48d6b3 to 5afe883 Compare October 3, 2016 15:57

danpovey merged commit 4b5eed2 into kaldi-asr:master Oct 4, 2016


		DropoutComponent(int32 dim, BaseFloat dp = 0.0) { Init(dim, dp); }

		DropoutComponent():dim_(0), dropout_proportion_(0.0) { }

Added Dropout component in netted to be used ephemeral connection. #1032

Added Dropout component in netted to be used ephemeral connection. #1032

Conversation

pegahgh commented Sep 17, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danpovey commented Sep 17, 2016

danpovey commented Sep 17, 2016

pegahgh commented Sep 17, 2016 • edited Loading

danpovey commented Sep 18, 2016

pegahgh commented Sep 18, 2016 • edited Loading

danpovey commented Sep 20, 2016

pegahgh commented Sep 21, 2016

danpovey commented Sep 21, 2016

pegahgh commented Sep 21, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pegahgh commented Sep 21, 2016

danpovey commented Sep 21, 2016

Choose a reason for hiding this comment

pegahgh commented Sep 25, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danpovey commented Sep 25, 2016

pegahgh commented Sep 26, 2016

danpovey commented Sep 26, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danpovey commented Sep 28, 2016

@vijayaditya commented on this pull request.

vijayaditya commented Sep 30, 2016

danpovey commented Sep 30, 2016

pegahgh commented Sep 30, 2016

pegahgh commented Sep 30, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

danpovey commented Oct 1, 2016

pegahgh commented Oct 3, 2016

pegahgh commented Sep 17, 2016 •

edited

Loading

pegahgh commented Sep 18, 2016 •

edited

Loading