This is our official implementation of CAT-Seg!
[arXiv] [Project] [HuggingFace Demo] [Segment Anything with CAT-Seg]
by Seokju Cho*, Heeseong Shin*, Sunghwan Hong, Seungjun An, Seungjun Lee, Anurag Arnab, Paul Hongsuck Seo, Seungryong Kim
We introduce cost aggregation to open-vocabulary semantic segmentation, which jointly aggregates both image and text modalities within the matching cost.
For further details and visualization results, please check out our paper and our project page.
We release the code for our interactive demo, where you can run the demo on your local or desired devices!
Please follow the original installation process below before getting started with the demo.
We use gradio, which can be installed as follows:
pip install gradio
For the demo, CAT-Seg (L) and SAM (ViT-H) is used as default. Please download each weights into the project directory.
The demo can be launched with the app.py
file.
python __init__.py [-- opts [OPTS]]
# For CPU usage
python __init__.py --opts MODEL.DEVICE "cpu" [OPTS]
Please follow installation.
Please follow dataset preperation.
We provide shell scripts for training and evaluation. run.sh
trains the model in default configuration and evaluates the model after training.
To train or evaluate the model in different environments, modify the given shell script and config files accordingly.
sh run.sh [CONFIG] [NUM_GPUS] [OUTPUT_DIR] [OPTS]
# For ViT-B variant
sh run.sh configs/vitb_r101_384.yaml 4 output/
# For ViT-L variant
sh run.sh configs/vitl_swinb_384.yaml 4 output/
# For ViT-H variant
sh run.sh configs/vitl_swinb_384.yaml 4 output/ MODEL.SEM_SEG_HEAD.CLIP_PRETRAINED "ViT-H" MODEL.SEM_SEG_HEAD.TEXT_GUIDANCE_DIM 1024
# For ViT-G variant
sh run.sh configs/vitl_swinb_384.yaml 4 output/ MODEL.SEM_SEG_HEAD.CLIP_PRETRAINED "ViT-G" MODEL.SEM_SEG_HEAD.TEXT_GUIDANCE_DIM 1280
eval.sh
automatically evaluates the model following our evaluation protocol, with weights in the output directory if not specified.
To individually run the model in different datasets, please refer to the commands in eval.sh
.
sh run.sh [CONFIG] [NUM_GPUS] [OUTPUT_DIR] [OPTS]
sh eval.sh configs/vitl_swinb_384.yaml 4 output/ MODEL.WEIGHTS path/to/weights.pth
We provide pretrained weights for our models reported in the paper. All of the models were evaluated with 4 NVIDIA RTX 3090 GPUs, and can be reproduced with the evaluation script above.
Name | Backbone | CLIP | A-847 | PC-459 | A-150 | PC-59 | PAS-20 | PAS-20b | Download |
---|---|---|---|---|---|---|---|---|---|
CAT-Seg (B) | R101 | ViT-B/16 | 8.9 | 16.6 | 27.2 | 57.5 | 93.7 | 78.3 | ckptĀ |
CAT-Seg (L) | Swin-B | ViT-L/14 | 11.4 | 20.4 | 31.5 | 62.0 | 96.6 | 81.8 | ckptĀ |
CAT-Seg (H) | Swin-B | ViT-H/14 | 13.1 | 20.1 | 34.4 | 61.2 | 96.7 | 80.2 | ckptĀ |
CAT-Seg (G) | Swin-B | ViT-G/14 | 14.1 | 21.4 | 36.2 | 61.5 | 97.1 | 81.4 | ckptĀ |
We would like to acknowledge the contributions of public projects, such as Zegformer, whose code has been utilized in this repository. We also thank Benedikt for finding an error in our inference code and evaluating CAT-Seg over various datasets!
@misc{cho2023catseg,
title={CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation},
author={Seokju Cho and Heeseong Shin and Sunghwan Hong and Seungjun An and Seungjun Lee and Anurag Arnab and Paul Hongsuck Seo and Seungryong Kim},
year={2023},
eprint={2303.11797},
archivePrefix={arXiv},
primaryClass={cs.CV}
}