There are a lot of OSS vector search databases out there, we could probably list the main ones:
- Qdrant: https://github.com/qdrant/qdrant
- Weaviate: https://github.com/weaviate/weaviate
- Milvus: https://github.com/milvus-io/milvus
What else?
Another one most similar to qdrant we track internally is lancedb. They are clever by supporting an embedded architecture, so an architectural reason to prefer over most existing OSS DBs. In our survey 2 years ago, we predicted specialized vector DBs having regular OSS DBs be the elephant in the room, and missed embedded as a fundamentally different category: https://gradientflow.com/the-vector-database-index/ .
(Good luck to qdrant! I'm happy they waited before raising, hopefully this means they can operate more healthily than otherwise and easier to maintain the discipline to do that!)
And then there are traditional databases and search products that are integrating vector search capabilities as well: Postgres, Elasticsearch, Opensearch, Solr.
They each have their limitations of course but the 28M round suggests a moat that I'm not seeing that clearly in terms of tech. What's so special about qdrant relative to their competition?
At least they are Apache licensed for now. So, that's nice. But that also means e.g. Apache Lucene could borrow some code from them to beef up their vector search capabilities. Which would benefit Elasticsearch, Opensearch, and Solr which all depend on Lucene.
Which raises the question what the point is of QDrant long term and why investors are betting on this as opposed to other things.
It seems to me that the main challenge with vector search is inference cost (at index and query time), not storing the vectors. A secondary concern is the vector comparisons at query time. A good way to cut down on that is to reduce the overall result set using traditional search or query mechanisms. In other words, you need
From the enterprise perspective, which of these vendors proved the best combination of security, availability, performance and pricing will matter. when we run benchmarks on our (self hosted) LLMs, we do not a clear idea of where we have bottlenecks and we end up assuming its the GPU/memory. And our pilot implementation will never go into production as the security model is nearly non existent in our implementations; the execs AND qa are getting the same RAG outputs. It is all very new to us and our teams. If a vendor can outperform its competition in our tests and show credible security model with segmentation of knowledge, that would be the choice.
I ask because this was the statement from the PP
There are a lot of OSS vector search databases out there, we could probably list the main ones
...
What else?The cases where I've seen this work are when the DB offers something way ahead of what their competitors offer. For example, KDB+ was historically unrivaled when it came to ultra high performance time series storage and Aerospike is very hard to beat for extremely high performance multi-node K/V.
Otherwise there's little to stop a larger company from offering the OSS competitor to your DB as a service for a lower cost and invest eng resources to close the gap.
I also like https://github.com/lancedb/lancedb
(discovered from sibling comment here: https://news.ycombinator.com/item?id=39103322)
I’ve been working with vectors for over a decade; particularly with embeddings used in AI. We’re talking projects from 100k to 100B+ records, used for AI applications
Postgres, particularly with pgvector and derivatives, can handle to millions of records very rapidly no problem. It’s very cheap, scales great, and is accurate.
I’m sure some of these open source solutions are improvements. That said, weigh vendor lock in, cost, risk and in the end it usually makes very little sense.
https://www.capitalone.com/tech/machine-learning/learning-em...
Running an R&D group in the department.
Those high profit margin part of the LLM business is for big players only, they don't burn hundreds of billions to offer you opportunties to cut their profit by capitalizing on their core business.
Communism doesn't exist in high tech. People don't work their xxx off to pave ways for your free lunch for life.
I can see them replacing it if Qdrant isn't able to scale to their needs - thats why we ended up with Dynamo, Spanner, MyRocks. However its likely that its probably easier to just acquire the team - like Apple did with Foundation - and become project stewards than trying to invent a new datastore to save pennies.
I know that, and you know that, but tell it to a jury of East Texas hayseeds.
it is also worth to mention that even if there is going to be a market called "vector databases", which is highly unlikely, you can't just written off all existing regular databases and pretend that they are not going to just walk in and take over.
all in all, there is no reason to believe it is a hot market. it is much better to ask is there going to be a market at all.
They are also a Rust shop.
Who says Germany has no cool startups.
EDIT: Yes, it was Grok.
What have your experiences with vector databases been? I've been using https://weaviate.io/ which works great, but just for little tech demos, so I'm not really sure how to compare one versus another or even what to look for really.
Once in a whole supabase slips into a project, but that's basically just Postgres with some bells and whistles on top.
I got nothing bad to say about Pinecone, We aviate, Chroma etc. but when it comes to dbs, I like to go with the devil I know.
for now is the keyword here.
When the company grow to 10k or 30k people, there will be teams competing for visibility, someone is going to build their inhouse "vector database" to get his/her slice of the pie. Do you still believe that any AI major player is going to reply on some external vector databases?
Small revolution indeed.
$ git grep "Arc<" | wc -l
451
I genuinely ask - there are a lot of other problems in the RAG, fine tuning, AI/LLM, retireval space, to solve. And more and more vector retrieval is, while not 100% solved, at least is something the community has a grasp on the tradeoffs. Solved to the point that squeezing a bit more recall out of vector retrieval isn't the problem anymore.
I think this is a bit of a strawman. I don't think recall is the main point these systems are trying to sell us on, it's more about robustness and ease of use compared to building something inhouse or using a lower level library to build a system on top of it just for this small part of your overall project/product (be it RAG, search, whatever).
I guess Lucene-based solutions, while very mature overall in terms of engineering, lagged behind this functionality (out of caution, trying to build what's going to be long term useful) and are also perceived a bit too cumbersome. So these stores do make sense, I think. The core functionality is nothing too complex (at least HNSW), but hiding it behind a stable black box with just a few inputs and levers, has value for people that are likely to use these stores.
There is still work to be done in vector databases. None of the products have perfected hybrid search yet, for example, and performance varies a lot between products; they are not fungible.
What do you think you milvus? https://milvus.io/. The difference seems significant from the architecture perspective.
Re embeddings, you would likely get better results if you train your own embeddings model. A popular approach is ColBERT, which anecdotally outperforms vector search in border cases[1]. Second is training an embedding model using initial layers of an LLM. [2]. In Colbert's case once it's trained, you dont need a db to store the vectors.
[1]: https://twitter.com/arjunkmrm/status/1744741903646773674 [2]: https://huggingface.co/intfloat/e5-mistral-7b-instruct
Support from the team is great however, and congrats to them for this round!
We use 768 dimensional vectors for our items with several other payload filters (e.g. language). Performance has been good and I think the qdrant team focus on the right features without creeping into other areas.
It's probably going to stay, but I'm also evaluating databricks new vector store as we're using databricks for all the analytics parts of the app already, and having them all on the same infrastructureis appealing.
https://techcrunch.com/2024/01/23/qdrant-open-source-vector-...
(now I'm gonna plug what I work on)
If you're interested in a more scalable vector database written in Go, check out Milvus (https://github.com/milvus-io/milvus)
Let’s see what they can do in a year or more with that new capital.
"We are getting many applications for this position. Usually, a test task would help preselect suitable candidates. However, since we develop open-source software, we rely on contribution.
You can build an open-source Qdrant connector to another framework or library. The simplest one would be, for example, a Streamlit data connector. But other ideas are more than welcome!
No limitations and no deadline. As long as this job position is online, we accept submissions. After you are done, send us an email to career@qdrant.com with the link to the repo. We will review it and get back to you asap."
No interviews, conversation before this email. Hope they see and fix this.
Edit : No Pay.
I think there are some things I wish were easier, for example finding and updating points, and the UI could be better.
exactly! if there is a real & strong demand, we'd be seeing open source ones get upgraded & ready in months. it is more like just one of those "I want to build something easy in the core but fancy in its name to get some quick VC $"
companies like openai will fill the gap by offering such features out of the box. there is no logical reason why and how a big AI tech giant is going to take all hard work and letting someone else to take the profit by ignore the last mile issue. in fact, openai has already released such APIs in their last devday event.
I don't even know any 10-30k people companies that build their own search, most I've known use elastic search or lucene.
It's had to imagine that a few of these vectordb companies don't establish themselves as the standard solution, being the equivalent to MongoDB in their space. The other competent players will very likely get acquired.
Certainly these vectordb companies are in a better long term standing that the bajillion companies rushing to build products that are just calling an API endpoint at the end of the day.
for your question, google built spanner (that was like 12 years ago) and leveldb when it had about 30k people. Meta/facebook forked leveldb and started building rocksdb when they had less than 30k people.
no one is saying any banking, insurance, retail companies with 30k employees should be building their own databases/search engines. They shouldn't. That is actually my main argument - those companies shouldn't get too involved for infra like that, they should just take the offerings of the big tech companies.
Curious if the co-founder who was posting in here will share his take.
For a job switch, I need to spend time in three different stages:
---------------
Preparation:
Leetcode (Blind 75) : 150 hours
System Design + DBMS + OS + Networking : 100 hours
Behavioural Questions (preparing STAR format answers): 10-20 hours
------------
Application:
Avg time for sending 500 applications: 20 hours (Assuming 1 application every 2.5 minutes)
-------------
Interviews:
Let's say I got 25 callbacks and 10 of them asked for takehome.
Person to person interviews time: 25 * 3 = 75 hours
Takehomes: 10 * 6 hours = 60 hours.
-------------
All in all, I'm already spending 415 hours of unpaid work to get x% of salary increment. Not including the side projects or hackathons we may need.
So having a takehome exercise asking to make an active contribution to the company is....bad. Sure i can reject it but not everyone will. which is what led us into the multiple rounds of algo interviews hellhole.
I apologize if what I'm saying is harsh. All I want is for leadership to see us as humans with families and not monkeys jumping through hoops.
If they're not then it's a scandal.
Like, you had to have actively developed mods for them, for free, for years, and be famous in the community, then they'll hire you (If you want).
This works because the working conditions there are far far better than your average game company. And probably much more fun than say a bank.
Over the years I've read/heard plenty of stories (here on HN and elsewhere) of people getting hired for their open source contributions to some stack that some company is using/developing.
So here I am willing to give some slack here to Qdrant. They get extremely qualified candidates who can jump right in, and candidates get told the rules of the game up front. It feels fine?
Surely much better than fake take home tests, whiteboard tests, leetcode onslaught, and 7 layers of interviews.
So if creating a high quality repo is 60-90% of your job interview that seems pretty good. As long as they are not ghosting high quality contributions that is.
I will change my view if they get 20 high quality connectors out of this and noone gets hired from that pool of candidates.
From this job posting [0]
> Compensated Interview Tasks: We value your time and effort; candidates will be compensated for completing interview tasks.
[0] -- https://join.com/companies/qdrant/10275180-core-rust-enginee...
From your perspective, you're filling out an application, maybe writing a cover letter, but on the other side, there are 100+ applications like yours. Not all of them are qualified, CVs are not a trustable source anyway.
That's why companies add tests to filter first, then interview later.
As a thumb rule, I'm happy to put 4x more effort than the company. If they interview me for 1 hour, I spend 4 hours doing the take-home. Anything more feels like exploitation.
HR gets paid to talk to candidates. I don't get paid to apply. The initial screening call is what allows a company to gauge the relevancy of a candidate. Let him speak about some of the topics and see how in-depth they go. Either the HR is familiar enough with the tech to understand proficiency (think a student listening to a maths professor) or they let a TL have a short conversation. I've overheard unqualified HR do their jobs badly, too; They laughed at picking them by looks and "feels". But, that's out of scope here.
A large company has millions to invest in different areas. Intrinsically, it has a much larger margin of error. You accidentally overprovisioned some resources and cost the company 10k? Tis but a scratch. You POC some personal project and accidentally get billed 10k? That is not the same.
A company can spend money on hiring. It is expected to. A private person can't spend money on applying to jobs. It isn't expected.
It's interesting to see how the shift goes from the self to the company [and to the country]. A little bit of communist propaganda goes a long way, eh, comrade NPC?
Vector databases are not about hosting LLMs or AI models, they are about storing and comparing embeddings vectors. You generate those with an AI model. OpenAI provides a few of those but GPT 4 is not typically what you'd use for this.
Model training and inference is typically not what a vector database does. You need it but to populate a vector database with content. Qdrant is not an exception to this, it uses third party models and inference technology for this (all the usual suspects basically). I just looked at their documentation to confirm this (but do correct me if I'm wrong).
Additionally, it lists all the classic use cases for vector search as its use cases (image search, semantic search, recommendations, similarity search, etc.). I'm sure it's awesome. But in the end it stores and compares vectors using an open source (and possibly patented?) algorithm. Which means if their approach is particularly good and novel, it will get copied in no time by other open source vector search capable products (i.e. most serious databases and search engines at this point). If it hasn't been already. I don't see dedicated vector database products having any inherent advantage here. Rather the opposite since they lack a lot of features that you might also need.
The real issue is actually Google Cloud or AWS having a vector search solution. Or companies having a Lucene-based search engine or existing DB for vector based search.
When someone calls others "npc", I understand that they are complete psychopaths that are somehow capable of thinking that the other people don't live the full human experience as they do.
I’m genuinely curious to know if it’s any good.
That means the lifetime of that object must be tracked at runtime, with a garbage collector. This is where you get Arc (or shared-ptr).
I'm not sure how you would safely solve this in C++, but FWIW, scylladb is a high performance database that also makes use of shared_ptr.
At a high level there are a few differences
- Control over embeddings. What gets embedded? What are the output vectors? What models do you use? How do you handle multimodal input?
- Performance. When you make a call to Assistants, you have to wait for the Assistant to understand that it needs to do RAG. This performance hit is actually quite large (look at the two videos on the blog for reference)
- Cost. OpenAI has an incentive to load the context window to consume more tokens. A few dozen calls to Assistants was costing me around $10.
Sorry, but I am not going to read it as it is not an apple to apple comparison atm. OpenAI just released its assistants APIs literally just weeks ago, when so called vector databases have been burning money for ages. You can write a thesis on how those vendors are doing slightly better for now, that won't be the big picture showing the reality on the ground. All those minor issues & unreasonable restrictions can be solved & removed, I don't see any real challenge for openai to implement them. Give openai a few months, they will convince most vector database vendors & gamblers to pack up and leave the field.
let me repeat what I have already explained - when compared to today's leading AI tech, a "vector database" is just ancient tech. major players are going to build their inhouse solution or they'd conclude it to be some kind of labor intensive & low profit margin baggage and outsource it.
you can build a business around it, just like all major tech companies have cleaning guys work for them one way or another, people have to realize that it doesn't make carpet cleaning a high tech or strategically important business.
There are a ton of open questions. If you think about Elasticsearch as a similar domain, you have complexity at the ingest, storage, and horizontal scalability layer. If you think places are going to invest in their own distributed system that handles these components, I think you’d be as wrong as saying that people will invest in their own managed Lucene implementations.
You are absolutely right into setting your own rules as well, those haven't overlapped.
Also, not all companies try to maximize for "top tier developer", it seems they are maximizing for "top motivated developer", which does not seem stupid either.
This is an open source database company, obviously the best candidates are contributors to the project first.
Picky ivy league graduate that farms leetcode and system design question is very very low in the ranking of what such companies should look for.
The trouble is that the leadership does see you as human, which results in them trying to say "Go away! You are not welcome here." as politely as possible.
Well this is the problem with tech hiring. Not convinced an IQ test or other “unstudyable” exam wouldn’t be better.
Hell maybe nepotism was the way to go all along. Do other industries require 10 weeks of full time practice to get a gig?
Most other jobs, including Mom & Pop Tech Co., are happy if anyone applies at all and will take what they can get.
And which was exactly my point.
> both Scylla and Redpanda, two high performance C++ databases both use shared_ptr
Such argument doesn't make sense since you will find usage of shared-ptrs in probably more or less every high performance C++ codebase out there. The secrete sauce is to know _where_ and _when_ to use it. I'm not against it but I'm definitely not in favor of language forcing me to user it when I don't need it.
The people I have met with the lowest tolerance for this stuff are the ones who actually produce the most impactful work. Partly because they don't do work that has no impact on their lives.
Edit: Obviously, they can do whatever they want, but that doesn't mean that it's a good sign from outside.
All it says is your hobby is programming.
2. The key to using a benchmark is not to find which vector database is the "best" but which one is most suitable for your use cases and applications. Actually, there is no "universally best" vector database for any situation. So I don't think anyone should be able to cheat on this.
No this is not vendor neutral unless you can prove that zillz included all tests that would expose its weakness.
> If you've used it or checked its leaderboard
see, you called it "leaderbord", it is a good proof that it is not vendor neutral. By using such highly biased tool, its hosting url has both the name of your company and the term "leader", it implies that the company is a leader when compared to other "vector" databases. tell me how such dirty trick is called vendor neutral?
please, HN is a place full of professional developers, many of those including myself have been in the business long enough to tell what is cheap propaganda and what is real cool tech.
> Actually, there is no "universally best" vector database for any situation.
because whether there should be a product category called "vector database" is in doubt in the first place. as explain, it is a low tech stuff significantly easier to design & implement than today's regular databases (e.g. cockroachdb). its role will eventually be filled by other real databases.
I won't argue with you about whether "vector database" should be a product category. You are not the first or even the last person who has doubts about it. Maybe many years later, we'll have the answer.