Search ArXiv Fluidly(searchthearxiv.com) |
Search ArXiv Fluidly(searchthearxiv.com) |
If this is your project, please talk about how you made it.
Code is on github https://github.com/augustwester/searchthearxiv
It depends on whether you do full text search, or abstract only. If you do full text, I’d guess about 1k tokens per page, 10 pages per paper? So that would be 3B tokens, which would cost you 60$ if you use the cheapest embedder.
If you just do abstracts, the costs will be negligible.
Plus you could use a mixed system: first you index the abstract of the most relevant 50 papers, then embedd the text of those 50 in order to asses which are truly relevant and/or meaningful.
I think doing it locally with an open source model would be a lot cheaper as well. Especially because they wouldn't have to keep using OpenAI's API for each new query.
Edit: I overlooked the about page (https://searchthearxiv.com/about), seems like they *are* using OpenAI's API, but they only have 300K papers indexed, use an older embedding model, and only calculate embeddings on the abstract. So this should be pretty cheap.
Searching the embeddings is a different problem, but there are lots of specialised databases that can make it efficient.