LeGR implementation

[mkaglins]

To run an experiment with LeGR algorithm "compression" config section schould looks like:

  "compression": {
        "algorithm": "filter_pruning",
        "params": {
            "pruning_flops_target": 0.3, // final FLOPs pruning target that will be fine-tuned
            "weight_importance": "legr",
            "all_weights": true,

            "prune_downsample_convs" : true, // recommended to allow pruning all nodes (algorithm potentially can decide pruning level for all nodes)
            "prune_first_conv": true, // recommended to allow pruning all nodes (algorithm potentially can decide pruning level for all nodes)
            "prune_last_conv": true, // recommended to allow pruning all nodes (algorithm potentially can decide pruning level for all nodes)
            "legr_params":
            {
                "generations" : 400, // Number of generations in Evolution algorithm (400 is default). Can be lower in simpler cases.
                "max_pruning": 0.8, // pruning level to train LeGR algorithm with. Can be equal to the target pruning level. In paper recommended to be a higher potential pruning rate for this model.
            }
        }
    } 

Settings in the "optimizer" section will be used as settings for the final fine-tuning optimizer.

[mkaglins]

@AlexKoff88 please, take a look.

[MaximProshin]

cc @AlexanderDokuchaev

[AlexKoff88]

General comments:

• We need to parallelize computations (model forward) in LeGR initialization step, namely validation of sampled models
• I also noticed that LeGR initialization output is dumped to nncf_output.log while most of the logs are presented in output.log. I wonder why do we need the former at all?

[lzrvch]

General comments:

• We need to parallelize computations (model forward) in LeGR initialization step, namely validation of sampled models
• I also noticed that LeGR initialization output is dumped to nncf_output.log while most of the logs are presented in output.log. I wonder why do we need the former at all?

One possible scenario for parallelization is to manually wrap the model with DataParallel during LeGR search inside the algo and unwrap it after initialization so that the user can utilize whatever training mode they want later on.

[vshampor]

General comment - use lowercase for all .py filenames, Windows won't be able to discern capitalizations anyway

[0de554K]

Jenkins please stop all builds

[0de554K]

Jenkins please retry a build

[0de554K]

Jenkins please stop all builds

[0de554K]

Jenkins please retry a build

[0de554K]

Jenkins please retry a build

[0de554K]

Jenkins please stop all builds

[0de554K]

Jenkins please retry a build

[mkaglins]

Jenkins please retry a build

[0de554K]

Jenkins please retry a build

[mkaglins]

Jenkins please retry a build

[mkaglins]

Jenkins please stop all builds

[mkaglins]

Ticket for documentation update: 59183
Ticket for tests of distribution callbacks in case of multiple-use: 59185

[mkaglins]

Results of reproducing this algo after all merges:

CIFAR-100:
Mobilenetv2 (CIFAR-100)
Original acc=68.61%

Algo\PR	0.4	0.6	0.8
FP + geomedian	68.279	61.388	54.647
LeGR (200)	68.149	68.980	68.109

Resnet-50 (CIFAR-100)
Original acc=78.51%

Algo\PR	0.4	0.6	0.8
FP+geomean	77.885	77.294	74.038
LeGR (200)	78.466	77.143	76.112

Imagenet:
Resnet-18, PR=30%:

Original	FP+geomean	LeGR
69.76	68.72	69.648

[MaximProshin]

@mkaglins , can you please update the table below by

• FP32 accuracy
• pruning rate for resnet-18 (I assume it's 30%)
  ?

[mkaglins]

@mkaglins , can you please update the table below by

• FP32 accuracy
• pruning rate for resnet-18 (I assume it's 30%)
  ?

Done.

[evgeniya-egupova]

(Issue 59470) @mkaglins, BTW, torch still expects weight_importance attribute