Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pruning masks applying #994

Merged
merged 3 commits into from
Nov 3, 2021
Merged

Pruning masks applying #994

merged 3 commits into from
Nov 3, 2021

Conversation

daniil-lyakhov
Copy link
Collaborator

Changes

##MKaglinslaya

  • FilterPruningBlock changed to apply masks to weights, not only store mask
  • Masks now applying only on the forward pass, all zeroing of original model weights (in apply_mask) was removed
  • To apply masks to Batch/Group Norm layers, these layers are wrapped as NNCF modules and FilterPruningBlock added to them. Masks propagation to these layers from convolutions.
  • UpdateWeightAndBiasPruning pre-layer operation was added to consistently prune weights and bias (@vshampor Please, take a look on these changes)
  • zero_grad attribute was deleted as not needed anymore
    @AlexanderDokuchaev @daniil-lyakhov Please, take a look at these changes in pruning.
    cc @evgeniya-egupova

##DLyakhov

  • Fix sparsity algo to prevent batchnorm layers sparsity
  • Remove batchnorm wrapping in pruning algo when batchnorm pruning is disable

Reason for changes

New masks applying mechanism allow re-pruning of the model since weights aren't pruning permanently.

Related tickets

Ticket ID 53524

Tests

A couple of tests in test_algo was changed to test the new mechanism (instead of zeroing weights zeroing of conv outputs are checked).

@daniil-lyakhov
Copy link
Collaborator Author

Unresolved #985 (comment)

@daniil-lyakhov daniil-lyakhov mentioned this pull request Oct 26, 2021
Closed
@daniil-lyakhov
Copy link
Collaborator Author

Successfully checked there is tf test which checks is batchnorm layers wrapped by NNCFWrapper after pruning algo: tests.tensorflow.pruning.test_algorithm.test_masks_in_concat_model

@daniil-lyakhov
Copy link
Collaborator Author

daniil-lyakhov commented Oct 27, 2021 •
edited
Loading

TODO:

  • Convert and check pt pruning checkpoints
  • Support backward compatibility for checkpoints

SOTA checkpoint build 599 603 604

@daniil-lyakhov
Copy link
Collaborator Author

SOTA check
607, new checkpoints - all checked
608, old checkpoints - all checked except pruning checkpoints, as expected
CC: @vshampor

@vshampor vshampor merged commit 0919f79 into openvinotoolkit:develop Nov 3, 2021
kshpv pushed a commit to kshpv/nncf that referenced this pull request Oct 11, 2022
@daniil-lyakhov
Pruning masks applying (openvinotoolkit#994)
9c3c075 
* Pruning masks applying

* Enable backward compatibility

* Backward compat test
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@alexsu52 alexsu52 alexsu52 approved these changes

@ljaljushkin ljaljushkin ljaljushkin approved these changes

@vshampor vshampor vshampor approved these changes

@evgeniya-egupova evgeniya-egupova evgeniya-egupova approved these changes

@AlexanderDokuchaev AlexanderDokuchaev Awaiting requested review from AlexanderDokuchaev

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@daniil-lyakhov @alexsu52 @ljaljushkin @vshampor @evgeniya-egupova