Graph Mining Library(github.com) |
Graph Mining Library(github.com) |
I guess it coincided with the social network phenomenon. Much more recently geometric learning (ML on graphs and other structures) shone, until LLMs stole their thunder. I still think geometric learning has a lot of life left in it, and I would like to see it gain popularity.
Then there are "graph algorithms" such as PageRank, graph centrality, and such. In a lot of those cases there is one edge type or a small number of edge cases.
There are some generic algorithms you can apply to graphs with many typerd edges edges such as the magic SPARQL pattern
?s1 ?p ?o .
?s2 ?p ?o .
The thing about graphs is, in general, they are amorphous and could have any structure (or lack of structure) at all which can be a disaster from a memory latency perspective. Specific graphs usually do have some structure with some locality. There was a time I was using that magic SPARQL pattern and wrote a program that would have taken 100 years to run and then repacked the data structures and discovered an approximation that let me run the calculation in 20 minutes.
Thus practitioners tend to be skeptical about general purpose graph processing libraries as you may very have a problem that I could code up a special-purpose answer to in less time than you'll spend fighting with the build system for that thing that runs 1000x faster.
----
If you really want to be fashionable though, arXiv today is just crammed with papers about "graph neural networks" that never seem to get hyped elsewhere. YOShInOn has made me a long queue of GNN papers to look at but I've only skimmed a few. A lot of articles say they can be applied to the text analysis problems I do but they don’t seem to really perform better than the system YOShInOn and I use so I haven’t been in a hurry to get into them.
A graph of typed pointers. As you likely know, the basic element of RDF is not “foo has a relationship with bar”, but “foo has a relationship with bar of type baz”.
Also, the types themselves can be part of relationships as in “baz has a relationship with quux of type foobar”
> The thing about graphs is, in general, they are amorphous and could have any structure (or lack of structure) at all which can be a disaster from a memory latency perspective
But that’s an implementation detail ;-)
In theory, the engine you use to store the graph could automatically optimize memory layout for both the data and the types of query that are run on it.
Practice is different.
> Thus practitioners tend to be skeptical about general purpose graph processing libraries
I am, too. I think the thing they’re mostly good for is producing PhD’s, both on the theory of querying them, ignoring performance, and on improving performance of implementations.
2. Geometric learning is the broader category that subsumes graph neural networks.
Integrations include:
* NetworkX -- https://networkx.org/
* DeepGraphLibrary -- https://www.dgl.ai/
* cuGraph (Rapids.ai Graph) -- https://docs.rapids.ai/api/cugraph/stable/
* PyG (PyTorch Geometric) -- https://pytorch-geometric.readthedocs.io/en/latest/
--
1: https://docs.arangodb.com/3.11/data-science/adapters/
2: https://github.com/arangodb/interactive_tutorials#machine-le...
bazel build //...
bazel test //...
bazel query //...
bazel run //in_memory/clustering:graph
ERROR: Cannot run target //in_memory/clustering:graph
Or do(better?) frameworks for the same function as this code already exist(maybe networkx?)?
What is Starland ?
It's interesting and deceptively simple at first.
Is "Graph Mining" so ubiquitous that people know what this is all about?
If you need docs just read the .h files, they have extensive comments. I’m sure they’ll add them or maybe, just maybe, you could write some to contribute.
This would have made some of my previous work much easier, it’s really nice to see google open source this.
curious if this is typical dev experience inside google..
Some code libraries had excellent docs (recordio, sstable, mapreduce). But yes, reading the header file was often the best place to start.
Reading the code, especially the header files, seems to be pretty standard as far as what I see in non-open source code. So, it’s been my typical dev experience, I’d say if you’re somewhere that has gleaming, easy to understand docs that are actually up to date with the code you all have too much time on your hands, but I serially work at startups that are running to market.
Either they rewrite git history or it took about 2 years to get approval on making this repo public.
So probably they just started the project two years ago, had aspiration to open source, and finally just did now. Some teams might publish earlier, some like to wait until it's had enough internal usage to prove it out.
$ cat $(find . -type f | grep -vE LICENSE\|README\|BUILD\|bazel\|git\|docs) | sort -u | wc -l
8360 unique lines scattered across more than 100 files. Good luck deciphering that in a single day!
By the way, the first issue in the repo is a "Request for a more verbose README", which I agree with.
https://github.com/google/graph-mining/blob/main/in_memory/c...
This, too:
https://github.com/google/graph-mining/blob/main/in_memory/s...
Same with most of the other files.
How is it usable? It's usable if you want to find date within lots and lots of data efficiently. That's kinda Google's thing. :-D
This in turn allows you to only build and use the 'package' you care about without having to build the whole repo in other projects. Continuing on the above example, if you only wanted to use the asynchronous_union_find.h header file in your project, somewhere in your WORKSPACE file, you add the graph-mining library using a git_repository rule (see WORKSPACE.bazel for examples), and in a cc_library rule in a BUILD file inside your project, you can add a `@graph-mining//in_memory/connected_components:asynchronous_union_find`. Then you can include it as a header elsewhere. Building your project then only builds that package and its dependencies, and not the entire graph-mining library.
edit: thanks to https://sdkman.io/ it's up and running. It wasn't _that_ bad after all.
Generally I'd expect companies to open source things when it's proven itself internally and they want to reap the benefits of open source:
- Make internal engineers happy - engineers like having their code released outside the bounds of their company
- Prestige, which can help with hiring
- External contributions (not even code necessarily, just feedback from people who are using it can be amazingly useful for improving the software)
- Ability to hire people in the future who already know important parts of your technical stack, and don't need internal training on it
- Externally produced resources that help people learn how to use the software (tutorials, community discussion forums etc)
If the software is no longer used internally, open sourcing it is MORE expensive - first you have to get through the whole process of ensuring the code you are releasing can be released (lots of auditing, legal reviews etc), and then you'll have to deal with a flow of support requests which you can either deal with (expensive, especially if your internal knowledge of the tool is atrophying) or ignore (expensive to your reputation).
If your open source project/protocol is the most popular, and you have the governance over it, then you decide where it goes. Chromium is open source, but Google controls it, and everyone who depends on it has to follow. If Chromium was not open source, maybe Firefox would be more popular, and Google would not have control over that.
> or ignore (expensive to your reputation).
I don't think that anything is expensive for Google. They can do whatever they want.
FYI, I work at Google.
so as i see it you have like three options if you are unhappy with that:
1. close the tab
2. dig into the impl and learn as you go
3. do 2 but also write docs
i just really believe i've covered literally all the cases that any reasonable (not whiney, not entitled) person would concede.
> the first issue in the repo is a "Request for a more verbose README", which I agree with.
posted today - do you think it might have something to do with this post we find ourselves convening on? i.e. no one was so bothered about a lack of docs until now?
edit:
i forgot actually something else you could do: email the author and ask nicely for some tips.
If you `bazel build //...`, you should get the compiled libs under `bazel-out/*fastbuild/bin/`.
I’ll also say I think graph algorithms are overrated, I mean you know the diameter of some graph: who cares? Physicists (like me back in the day) are notorious for plotting some statistics on log-log paper, seeing that the points sorta kinda fall on a line if you squint and decide that three of the points are really bug splats and then yelling “UNIVERSIALITY” and sending it to Physical Review E but the only thing that is universal is that none of them have ever heard of a statistical estimator or a significance test for power law distributions. Node 7741 is the “most central” node, but does that make a difference? Maybe if you kill the top 1% central nodes that will disrupt the terrorist network but for most of us I don’t see high quality insights coming out of graph algorithms most of the time.
For people who've missed it: So You Think You Have a Power Law — Well Isn't That Special? (http://bactra.org/weblog/491.html) :)
Open source at Google generally takes the form of libraries rather than products. Often, that's something that an individual engineer is working on, and it's easier to open source than get the copyright reassigned (since Google by default owns any code you write). There are also libraries that are open sourced for business reasons - e.g. SDKs. You can tell the difference, because most individually-driven libraries contain the copy "Not an official Google product" in the README.
I found one after some searching: Nomulus. https://opensource.googleblog.com/2016/10/introducing-nomulu...
Go look at the manuals for IBM's Parallel Sysplex for mainframes and compare the simplicity of that to K8S for instance.
Or for that matter look at DCOM and the family of systems which Microsoft built around it which are truly atrocious but look like a model of simplicity compared to gRPC. (At least Don Box wrote a really great book about COM that showed people in the Microsoftsphere how to write good documentation.)
Or for that matter try doing something with AWS, Azure or some off-brand cloud and Google Cloud from zero (no account) and time yourself with a stopwatch. Well, it will be a stopwatch for AWS but you will probably need a calendar for Google Cloud.
10 mins spent on readme with some high level details is investment with 100x return for lib users.
Or just write a make file and cut all the bazel build optimization out.
They don’t put instructions on how to start a F1 car inside the cockpit, you don’t hop into a fighter jet and look for the push to start easy button, it’s expected that when you’re at that level you bring expertise.
But seriously, there is such a thing as industrialization. Google is notorious though for hiring 180 IQ people and getting them to perform at a 35 IQ level because there the documentation makes no sense, a procedure which should be done in one step really takes 300, fighting with C++, etc. They can afford to do it because it is just so profitable to help that guy who shows up on every. single. video. roll. who says “you can’t lose weight by exercising”, “you can’t lose weight by cutting carbs” who links to some video that drones on hours and hours and signs you up for some subscription you can never cancel to sell scam supplements.
Shame on them.
BTW, with high-complexity software there is enough question that you got it working right that you expect a significant process of testing that it works for your application. For instance if you got a hydrocode for simulating the explosion of a nuclear weapon you would not take for granted that you had built it properly and were using it properly until you'd done a lot of validation work. A system like that isn't a product unless it comes with that validation suite. The same is true for automated trading software (gonna hook it up straight to the market without validation? hope you have $100M to burn!)
... now there was that time a really famous CS professor e-mailed me a short C program that was said to do something remarkable that crashed before it even got into main() which did teach me a thing about C but that's not what a professional programmer does.
Its just frustrating that all the comments about an interesting library seem to be customer service complaints from people who never need to reach for this library. I was hoping for a real discussion, something I could learn from.
Now the Lotus notes patents have been long expired so I’d like to see some graph database based products that can do what Notes did 30 years ago but it is lost technology like the pyramids, stonehenge and how to make HTML form applications without React.
Nope, "of type baz" is not required.
There is also the option of blank nodes for objects, though in almost all implementations they are stand-ins for anonymous IRIs, so in some sense or another almost anything has "a" type.
?s ?p ?o .
?s <http://www.w3.org/1999/02/22-rdf-syntax-ns#type> ?type .
?s a ?type .
?p a rdf:Property .
?p rdfs:subPropertyOf :SomeSpecialKindOfProperty .(I’d consider myself qualified to evaluate it.. If I put two weeks into futzing with it.)
Every open source release I’ve done that’s been successful has involved me spending almost as much time in documentation, packaging and fit-and-finish work as I did getting working it well enough for me. It’s why I dread the thought of an open source YOShInOn as much as I get asked for it.
Sometimes though it is just a bitch. I have some image curation projects and was thinking of setting up some “booru” software and found there wasn’t much out there that was easy to install because there are so many moving parts and figured I’d go for the motherf4r of them all because at least the docker compose here is finite
https://github.com/danbooru/danbooru
even if it means downloading 345TB of images over my DSL connection.