The advent of text-image models, most notably CLIP, has significantly transformed the landscape of information retrieval. These models enable the fusion of various modalities, such as text and images. One significant outcome of CLIP is its capability to allow users to search for images using text as a query, as well as vice versa. This is achieved via a joint embedding of images and text data that can, for instance, be used to search for similar items. Despite efficient query processing techniques such as approximate nearest neighbor search, the results may lack precision and completeness. We introduce CLIP-Branches, a novel text-image search engine built upon the CLIP architecture. Our approach enhances traditional text-image search engines by incorporating an interactive fine-tuning phase, which allows the user to further concretize the search query by iteratively defining positive and negative examples. Our framework involves training a classification model given the additional user feedback and essentially outputs all positively classified instances of the entire data catalog. By building upon recent techniques, this inference phase, however, is not implemented by scanning the entire data catalog, but by employing efficient index structures pre-built for the data. Our results show that the fine-tuned results can improve the initial search outputs in terms of relevance and accuracy while maintaining swift response times.
Demonstration of CLIP-Branches search workflow. The user initiates a search with a query string and receives top k-initial results. These results are then labeled as positive (green) or negative (red) based on user preference, guiding the fine-tuning of the search. The fine-tuned results reflect more completeness and higher accuracy.
Search process of CLIP-Branches. Traditional text-to-image search engines only consist of the steps ❶ to ❸ while CLIP-Branches adds a fine-tuning stage to refine the initial search results (Steps ❹-❼). Index structures are employed during the search for faster execution.
A demo how to use CLIP-Branches can be found in the following video:
Come on and try it out yourself via the following link.
References
2024
CLIP-Branches: Interactive Fine-Tuning for Text-Image Retrieval
In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval (Demo track), SIGIR (CORE Rank A*) , 2024
The advent of text-image models, most notably CLIP, has significantly transformed the landscape of information retrieval. These models enable the fusion of various modalities, such as text and images. One significant outcome of CLIP is its capability to allow users to search for images using text as a query, as well as vice versa. This is achieved via a joint embedding of images and text data that can, for instance, be used to search for similar items. Despite efficient query processing techniques such as approximate nearest neighbor search, the results may lack precision and completeness. We introduce CLIP-Branches, a novel text-image search engine built upon the CLIP architecture. Our approach enhances traditional text-image search engines by incorporating an interactive fine-tuning phase, which allows the user to further concretize the search query by iteratively defining positive and negative examples. Our framework involves training a classification model given the additional user feedback and essentially outputs all positively classified instances of the entire data catalog. By building upon recent techniques, this inference phase, however, is not implemented by scanning the entire data catalog, but by employing efficient index structures pre-built for the data. Our results show that the fine-tuned results can improve the initial search outputs in terms of relevance and accuracy while maintaining swift response times.
@inproceedings{sigir,author={L\"{u}lf, Christian and Martins, Denis Mayr Lima and Salles, Marcos Antonio Vaz and Zhou, Yongluan and Gieseke, Fabian},title={{CLIP-Branches}: Interactive Fine-Tuning for Text-Image Retrieval},year={2024},booktitle={Proceedings of the 47th International {ACM} {SIGIR} Conference on
Research and Development in Information Retrieval (Demo track), {SIGIR}},note={Accepted (In press)},}
