I wrote this tool to get familiar with CLIP model, I know many people have written similar tools with CLIP before, but I'm new to machine learning and writing a classic tool helps my study.
The unusual thing with my version is, it is in pure Node.js, with the power of node-mlx, a Node.js machine learning framework.
The repo in the link is mostly about implementing indexing and CLI, the code of the model implementation lives as a Node.js module: https://github.com/frost-beta/clip .
Hope this helps other learners!
Show HN: Sisi – Semantic Image Search CLI tool, locally without third party APIs | Dark Hacker News
I was planning to do this myself lol. I was going to use SQLite as the index, and use `sqlite-vec` or something similar to query for similar files directly. I think the only other thing I was planning were more filters, `"positive term" -"negative term"` to be able to negate results, `>90"search"` to find images that match by >90% and some generic filters like `--size >1mb` to help narrow it down when you are looking for a specific image. Quantizing embeddings to make them smaller/faster also seemed interesting but I haven't tried doing it yet.
This script doesn't do search, but it generates keywords for images and places them in the image metadata. You can then search for keywords using something like Diffractor. I will warn though that any AI solution not geared towards NSFW will not give good information on NSFW images, though it may give a keyword such as 'intimate' or 'adult content' which is all you need.
How does CLIP compare to YOLO[1]? I haven't looked into image classification/object recognition for a while, but I remember that YOLO was quite good was working on realtime video too.
CLIP and YOLO work completely differently and have different purposes. CLIP uses transformers and embeddings and can compare text with images for classification. YOLO using a CNN and is trained with bounding boxes on images and is used for image recognition.
Give an image to CLIP and you can compare the similarity between the image and a sentence like 'a vase with roses in it'. Whereas with YOLO you give it an image and get the coordinates of bounding boxes around a vase, and around roses.
We have not decided what to do with it yet. It could be free, paid, or open source. However, the logic code for using semantic search with CLIP-compatible models on Android will be available on GitHub.
In this program CLIP is being used to create embeddings. A multimodal LLM does something very similar. In this case the language model is not needed because the embeddings are being used to search directly.
It’s not that straightforward due to spelling. Does that catch køk? Tihts? P. Nus? For a non English swear word, I had to ask 3 times and about a specific language to finally make that connection.
I read about a company in the 1990s that did that. They went one step further - picking culturally appropriate colors, shapes, numbers, and then permuting the brand names to favorable variations for a country. My (probably wrong) 25 year old recollection was when they introduced subway in China they basically found a way to pronounce it that translated to "this place is delicious". I bet it was in Wired. If not that, probably New York Magazine.
Yeah I noticed it was pretty shaky, change the prompt a bit and the result changes a lot. Not very reliable after all by itself, but used in conjunction with other methods.