Abstract

Imagine a designer browsing through an enormous image database for photos that contain both "a chair and a table" or a wildlife scientist attempting to find all photos of "brown bears near water." Locating such specific combinations manually is too time-consuming and cumbersome. To address this issue. This system SAM-CLIP Search is an area-based vision-language image search platform that incorporates CLIP and vision-language embeddings into the Segment Anything Model (SAM) to offer flexible prompt-based segmentation. Our approach makes precise picture search with point, box, or text prompts feasible compared to typical CBIR approaches, which often struggle with multi-object queries as well as cross-modal alignment. We propose a Ranking Optimization Layer (ROL) that produces context-specific relevance scores by aggregating spatial overlap (IoU) with semantic embedding distance and we substitute the conventional FAISS indexing with a light-weight cosine similarity approach to improve efficiency. Our method yields semantically and visually coherent matches on the COCO (val2017), Flickr30K and Fashion200K benchmarks. With maintaining fast inference, SAM-CLIP Search is better in critical metrics such as Recall@K, mAP and NDCG compared to baselines such as ViT+KNN, Deep Image Retrieval (DIR)and CLIP-only models. Its suitability for high-impact applications such as content curation, surveillance and medical image analysis is exemplified by a user study that verifies its effectiveness in difficult scene searches. The SAM-CLIP Search model achieves a retrieval accuracy of 92% with Recall@5 of 89.3%, Precision@5 of 87.1% and an average top-1 similarity score of 0.78 which makes SAM- CLIP Search Faster Region-Based Image Similarity Matching Using Lightweight Segmentation and Contrastive Learning a more accurate and efficient approach for region-based image retrieval.