Show HN: CozoDB, Hybrid Relational-Graph-Vector Database

Show HN: CozoDB, Hybrid Relational-Graph-Vector Database(docs.cozodb.org)

137 points by zh217 3 years ago | 51 comments

Hi HN! We're thrilled to share CozoDB v0.6, a monumental update to our FOSS database, which already unifies relational and graph features. With the addition of vector search, CozoDB becomes an even better companion for LLMs like ChatGPT.

This release introduces vector search using HNSW indices within Datalog, enabling seamless integration with powerful features such as ad-hoc joins, recursive Datalog, and classical whole-graph algorithms. This update significantly broadens CozoDB's capabilities.

Check out the linked release note for an in-depth look at the new features, comparisons to other systems, and intriguing AI development possibilities. We'd love for you to take a look! I'll be here to answer any questions you might have.

Looking forward to your feedback!

iFire 3 years ago |

Link to the github repository. https://github.com/cozodb/cozo

Rust and MPL2.0 License. https://github.com/cozodb/cozo/blob/main/LICENSE.txt

samsquire 3 years ago |

Wow this is awesome.

I like the notes about automated linking between notes in knowledge tools.

One of my ideas is to represent software architecture and system architecture as vector embeddings and transform the architecture dynamically.

Can scale from one box to internet levels of traffic with a Markov chain prompt

Edit: To clarify, I want to be capable of transforming existing software architecture with a prompt, that perhaps describes attributes of the system or describes capabilities.

nerdyadventurer 3 years ago | |

> I like the notes about automated linking between notes in knowledge tools.

May I ask where did you read about this? I am also interested.

samsquire 3 years ago | | |

This section.

https://docs.cozodb.org/en/latest/releases/v0.6.html#knowled...

luizfelberti 3 years ago |

How well does Cozo handle larger than RAM data? Does it only need to keep the query's answer set in memory?

By larger than RAM I mean the entire WikiData knowledge graph (~100GB), with something like 16GB of RAM.

Another question: any plans for supporting Parquet files with query pushdown? I honestly doubt Parquet's efficiency can be matched with RocksDB (but I'm happy to be proven wrong), and having to convert big datasets is always a pain...

zh217 3 years ago | |

For the parquet question: currently CozoDB is developed by a single developer (me), but I am starting to explore ways of expanding the development team. Certainly a lot more features will be added if that happens, and parquet support looks like a really useful one.

anonzzzies 3 years ago | | |

Any contact info for you to discuss contributing?

zh217 3 years ago | |

Yes this is correct, only the query's answer set need to be in memory. We are also working on streaming for the Rust API, in which case you don't even need to keep the whole set in memory for simple queries.

FYI here is a not very rigourous performance and memory usage analysis (for a previous version without the vector search capability): https://docs.cozodb.org/en/latest/releases/v0.3.html

luizfelberti 3 years ago | | |

Thanks for the speedy response!

Cozo is looking like a top-contender for my project so far :)

canadiantim 3 years ago |

Would using Cozo make sense for a social network? Certainly would make modelling comments easier but how well does it handle many concurrent users?

Can you intermix different storage engines as well? So e.g. a user could have a personal storage using sqlite but also easily save to a rocksDB storage as well?

In regards to the timetravel capabilities, can this be leveraged to implement git-like features querying these historical points in time in the data?

Also just curious your thoughts on how secure data is within Cozo? Or asked another way, how production-ready is Cozo. I know it's still early days but could Cozo be used as the primary database in a product being delivered today?

Great work all around, really awesome to see!

pizza 3 years ago |

Nice, I think you've just invented the first cellular sheaf db that I'm aware of

esafak 3 years ago | |

Can you explain?

pizza 3 years ago | | |

as I understand it a cellular sheaf complex assigns vector spaces to nodes and linear transformations to edges

zh217 3 years ago | |

Wow, cellular sheaves, that's a connection I haven't thought of before!

dang 3 years ago |

Show HN: Cozo – new Graph DB with Datalog, embedded like SQLite - https://news.ycombinator.com/item?id=33518320 - Nov 2022 (67 comments)

vlmutolo 3 years ago |

Well this all looks incredibly cool, and certainly beyond my current understanding of graph and vector databases.

You mentioned in the release that you wrote your own knowledge management tool. Is that published somewhere?

zh217 3 years ago | |

Not for the moment, it is not polished enough. Right now it is just a webapp written in React and prosemirror running on top of a CozoDB instance. And it is very rough around the edges (good enough for myself, but maybe not for others).

Once local LLMs that are powerful enough become available, though, I think I will try to find time to polish and publish it, since it can then act as a showcase for what a thinking agent can achieve.

lupickup 3 years ago | | |

How are you modeling the notes cozodb? I'd interested in parsing my Obsidian nodes into cozoDB as a way to setup alternate views on them. Curious how you thought about storing the bullets and relations between them.

srhtftw 3 years ago |

Wondering how it compares to rel¹.

¹- https://docs.relational.ai/rel/primer/basic-syntax

zh217 3 years ago | |

I'm not too familiar with rel, but from what I read, rel seems to be cloud only, and CozoDB always aims to be local-first. Another difference is that CozoDB has many whole-graph algorithms, and more can be added from user code (for example using Python), which I don't see rel having.

joshspankit 3 years ago |

Apologies for kind-of sidetracking, but I went back and read the blog post about Cozo’s time travel feature and wanted to add this approach: since relationships are cheap, you can have a “current status” relationship and multiple “past and current status” relationships. When the user updates their status it replaces the “current status” while also adding itself as a “past and current status”.

That way querying for the current status is a 1:1 graph lookup and you can reserve timestamp lookups for querying past statuses.

blatant303 3 years ago |

Someone should try feeding those conceptual maps back to a multimodal agent as pictures.

Also, to which extent is this related to quantum categorial grammars for NLP ?

ssivark 3 years ago |

If you want folks playing with integrating CozoDB and LLMs, it might be worth adding a CozoDB wrapper to Langchain :-)

zh217 3 years ago | |

Thanks for the suggestion--will surely do that!

freezed88 3 years ago | | |

would love to have CozoDB be a part of llamaindex too! have a bunch of integrations with existing vector db's https://github.com/jerryjliu/llama_index/tree/main/gpt_index...

vecrvecr 3 years ago |

I created this account to thank you for your work, this seems perfect for what I 'm working on! I'm about to finish the tutorial, it's very well done :)

vecrvecr 3 years ago | |

I've come back to say that I'm truly in awe at what you're doing. Given your examples (both in the tutorial and other articles), I think we might be thinking very similar things, but you're way ahead. You're amazing!

zh217 3 years ago | |

Thanks! I'm really glad that you find CozoDB useful!

joshspankit 3 years ago |

How do you handle different LLMs having different vector spaces?

zh217 3 years ago | |

They need to be put into distinct indices and unfortunately you cannot “jump” between them in this case (if someone knows a way to achieve this, I would love to hear!)

joshspankit 3 years ago | | |

I think I’m understanding that the item’s vector in one LLM can be stored as one index and the vector in another LLM can be stored as a second index without them colliding or one having to overwrite the other.

Is that right?

assface 3 years ago |

Is it called "CozoDB" or Cozo? Your README is inconsistent.

zh217 3 years ago | |

Sorry about that ... I will revise it to be more consistent. Cozo is a bit ambiguous, so now it is usually called CozoDB.

avinassh 3 years ago |

What is a vector database? and what makes better for LLMs?

zh217 3 years ago | |

The linked article explains these in details.

robcomptec 3 years ago |

Does this allow commit/rollback on a graph DB?

marginalia_nu 3 years ago |

Neat!

zh217 3 years ago | |

Thank you!