Zero-shot image classification is a computer vision task to classify images into one of several classes, without any prior training or knowledge of the classes.
In this tutorial, you will use OpenAI CLIP model to perform zero-shot image classification.
This tutorial demonstrates how to perform zero-shot image classification using the open-source CLIP model. CLIP is a multi-modal vision and language model. It can be instructed in natural language to predict the most relevant text snippet, given an image, without directly optimizing for the task. According to the paper, CLIP matches the performance of the original ResNet50 on ImageNet “zero-shot” without using any of the original 1.28M labeled examples, overcoming several major challenges in computer vision. You can find more information about this model in the research paper, OpenAI blog, model card and GitHub repository.
This folder contains notebook that show how to convert and quantize model with OpenVINO and NNCF
The notebook contains the following steps:
- Download the model.
- Instantiate the PyTorch model.
- Convert model to OpenVINO IR, using the model conversion API.
- Run CLIP with OpenVINO.
- Quantize the converted model with NNCF.
- Check the quantized model inference result.
- Compare model size of converted and quantized models.
- Compare performance of converted and quantized models.
- Launch interactive demo
We will use CLIP model for zero-shot image classification. The result of model work demonstrated on the image below
This is a self-contained example that relies solely on its own code.
We recommend running the notebook in a virtual environment. You only need a Jupyter server to start.
For details, please refer to Installation Guide..
