Mkaglins/pruning masks applying #985
Conversation
mkaglins
requested review from
a team,
vshampor,
AlexanderDokuchaev and
daniil-lyakhov
|
|
||
|
|
||
| def _conv_forward_proxy(self, input_, weight, padding_value): | ||
| def _conv_forward_proxy(self, input_, weight, bias, padding_value): |
There was a problem hiding this comment.
Is it really need to use proxy_weight and proxy_bias in _custom_forward_fn, instead of self in _conv_forward_proxy?
There was a problem hiding this comment.
Interesting question, I will discuss it with @vshampor.
There was a problem hiding this comment.
I think it's really needed, otherwise it would take original weight/bias directly from conv module, not from the proxy one (ProxyModule) with compressed params
| # 2. Set the masks for Batch/Group Norms | ||
| types_to_apply_mask = ['group_norm'] | ||
| if self.prune_batch_norms: | ||
| types_to_apply_mask.append('batch_norm') |
There was a problem hiding this comment.
Can we keep nodes we wrapped during _prune_weights , for example, in a way conv layers and bn / group norms are kept separately? Then here we can just take this nodes by id, for example, and update the masks after mask propagation algo
There was a problem hiding this comment.
Actually, Batch/Group Norm nodes are already stored separately in self._pruned_norms_operators.
So potentially we can iterate over self._pruned_norms_operators and update only these nodes.
There was a problem hiding this comment.
@daniil-lyakhov do you think it will be the clearer solution?
…also deleted as not needed anymore
Co-authored-by: Daniil Lyakhov <daniil.lyakhov@intel.com>
Co-authored-by: Daniil Lyakhov <daniil.lyakhov@intel.com>
|
Moved to #994 |
Changes
FilterPruningBlockchanged to apply masks to weights, not only store maskFilterPruningBlockadded to them. Masks propagation to these layers from convolutions.UpdateWeightAndBiasPruningpre-layer operation was added to consistently prune weights and bias (@vshampor Please, take a look on these changes)zero_gradattribute was deleted as not needed anymore@AlexanderDokuchaev @daniil-lyakhov Please, take a look at these changes in pruning.
cc @evgeniya-egupova
Reason for changes
New masks applying mechanism allow re-pruning of the model since weights aren't pruning permanently.
Related tickets
Ticket ID 53524
Tests
A couple of tests in test_algo was changed to test the new mechanism (instead of zeroing weights zeroing of conv outputs are checked).