FastSal

FastSal is a fast saliency map predictor base on MobielNet V2 backbone. FastSal (A) achieves near real time speed on CPU (>30FPS). There are 2 versions of FastSal, A(ddition) and C(oncatenetion), which can be chosen by specifying -model_type. Two sets of pretrained weights from the SALICON 2017 and COCO 2019 detection tasks are provided.

Publication

Find our pre-print on arXiv: FastSal: a Computationally Efficient Network for Visual Saliency Prediction.

@article{hu2020fastsal,
  title={FastSal: a Computationally Efficient Network for Visual Saliency Prediction},
  author={Feiyan Hu and Kevin McGuinness},
  journal={arXiv preprint arXiv:2008.11151},
  year={2020}
}

Install Dependencies

Use the package manager pip to install python library dependencies.

pip install -r requirements.txt

Usage

Download pretrained teacher models and/or weights

The SALGAN teacher weights can be downloaded from here.

The DeepGaze II computational graph can be downloaded from here

Pretraining

The SALGAN teacher weights are needed for pretraining.

Supports salicon or coco for -dataset_name. Supports A or C for -model_type.

Pretrain on SALICON 2017:

python pretrain.py \
-batch_size 30 \
-dataset_name salicon \
-dataset_path DATASET_PATH \
-teacher_path SALGAN_PATH \
-save_dir checkpoint \
-model_name pretrained_salicon

Pretrain on COCO:

python pretrain.py \
-batch_size 30 \
-dataset_name coco \
-dataset_path DATASET_PATH \
-teacher_path SALGAN_PATH \
-save_dir checkpoint \
-model_name pretrained_coco

Finetuning

The SALGAN teacher weights are needed for fine tuning. Fine tuning can be either from pretrained weights or random initialized weights. If -pretrain_path is not provided then randomly intialized weights are used.

Supports salicon or coco for -dataset_name. If -dataset_name is coco, -pseudo_path needs to be specified. Supports A or C for -model_type

Fine tune FastSal A with a SALGAN teacher and pretrained weights from SALICON 2017:

python fine_tune.py \
-model_type A \
-batch_size 30 \
-dataset_name salicon \
-dataset_path DATASET_PATH \
-teacher_path SALGAN_PATH \
-save_dir checkpoint \
-model_name finetune_salicon \
-pretrain_path weights/salicon_salgan_pretrain.pth

Fine tune FastSal C with pseudo saliency maps generated by DeepGaze II and pretrained weights from COCO. In order to train on the COCO dataset, pseudo labels have to be generated and saved at PSEUDO_PATH. In our example, we use Deepgaze II to generate pseudo saliency maps. We save the probability saliency map with central bias in numpy files. We can provide this data upon request.

python fine_tune.py \
-model_type C \
-batch_size 30 \
-dataset_name coco \
-dataset_path DATASET_PATH \
-save_dir checkpoint \
-model_name finetune_coco \
-pretrain_path weights/coco_salgan_pretrain.pth \
-pseudo_path PSEUDO_PATH

Evaluation

Supports salicon or mit1003 for -dataset_name. If mit1003 is chosen for -dataset_name, -batch_size needs to be set 1. Supports A or C for -model_type

Evaluation of FastSal A on the SALICON 2017 validation set:

python eval.py \
-model_type A \
-batch_size 30 \
-dataset_name salicon \
-dataset_path DATASET_PATH \
-model_path weights/salicon_A.pth

To load FastSal in python

Examples of how to load FastSal are in simple_load.py:

model = fastsal.fastsal(pretrain_mode=False, model_type='A')
state_dict, opt_state = load_weight(MODEL_PATH, remove_decoder=False)
model.load_state_dict(state_dict)

To predict using a image or a folder with images

In this example, the input and output folder are both image_example. Currently the script only takes images files with '.jpg', '.png' and '.jpeg' extensions.

python fastSal_predict.py \
-input_path image_examples/
-output_path image_examples/

The default settings for prediction -model_type is A, finetune_dataset is salicon, -batch_size is 1, prebability_output is False and gpu is True. Supports A or C for -model_type. Supports salicon or coco for -fintune_dataset. If -probability_output is True, the output files are saved in numpy data format.

The other arguments are:

python fastSal_predict.py \
-model_type A \
-finetune_dataset coco \
-input_path image_examples/ \
-output_path image_examples/ \
-batch_size 30 \
-probability_output True \
-gpu False

Future Development

TensorFlow implementation.

Acknowledgement

This work has emanated from research conducted with the financial support of Science Foundation Ireland (SFI) under grant number SFI/15/SIRG/3283 and SFI/12/RC/2289 P2.