Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix

Ontology-Driven Observability: Building an End-to-End Knowledge Graph at Netflix(netflixtechblog.com)

94 points by tosh 3 hours ago | 20 comments

smartmic 56 minutes ago |

I am in an enterprise which experiments with ontologies and knowledge graphs since its hype heydays ca. 15 years ago. It never took off, good ol’ relational databases and its modern day implementations in the cloud still rule the data landscape. Many smart consultants tried their best and we spend many millions over time into it. I would love to read more success stories, but they seem to be rare beyond academic exercises.

dhenneberger 22 minutes ago | |

95% of sparql, cypher, and gremlin can be lowered to sql anyway. I rolled by own sql lowering system for duckdb over iceberg which lets me do everything in that blog post but with no bespoke data layer.

1970-01-01 1 hour ago |

Without actual examples of it working to reduce latency, this is purely academic and not engineering. What happened to the 1.5s problem in the introduction?

CharlieDigital 55 minutes ago | |

I noticed this with a lot of blog posts from eng teams on AI.

Talk about what they built, not the business problem it solved and how it created value somewhere.

nout 37 minutes ago |

This seems somewhat outdated approach.

Agreed that centralizing identity is critical (which implies strong reconciliation systems, that can both do well on merges/splits/identity-shifts, but also need to enable backtracing to sources for each triple), but then I don't think it makes sense anymore to force ontology across all systems in the pipeline. As long as you give AI ability to do toolcalls/skills for each individual system and let it build up memory, then AI does much better job piecing together the actual practical ontology. You can give the observability AI to ask each system about it's state for the given piece of information and the AI should be free to make any toolcalls (e.g. call the specific system appropriately via CLI) it needs to gather the precise state.

I think human establishing the ontology gets in a way here, just let the AI poke the systems on its own via clean small interfaces.

smithclay 34 minutes ago |

It's at the end of this Netflix post: but the end game is to connect whatever your observability ontology is with whatever system the business people and data engineering teams use to be able to answer previously hard questions like "how much money did this operational or engineering change make or lose?"

In theory, at least.

Observability vendors have been beating the drum for 10+ years that there's so much untapped business value in operational data. Seems like we might be getting close to seeing how real that is.

threecheese 57 minutes ago |

The problem here is we're going to let the machines dictate the ontology, and we will lose expressiveness by reverting to the mean. All in order to communicate more cheaply with the machines - but on their terms.

Forgive the anthropomorphizing, it was necessary - "terms" has double meaning there, and each of us can understand that without an ontology - because we can access nuance. I can read schema, structure, and shape, and understand; but only one of those triggered some readers.

Anyway, the machines will choose terms which are most well-represented in the training data. This will normalize communication by chopping off the top-end where nuance lies. This is token-efficient.

Ontologies, in their implementation in the Linked Data ecosystem, do offer the ability to relate one understanding-system to another, but this is not token-efficient. I would love to see each domain, each business even, define their view of the world in the most expressive and culture matching owl representation, but I don't see it happening. "Shape" it is.

j45 1 hour ago |

I would welcome any and all links or resources about learning more about ontologies, including anyone's personal practice of/with them.

nico 1 hour ago |

Is this like XML and UML all over again?

llmslave 26 minutes ago |

Are knowledge graphs still relevant? Agents can grep and fetch on a unix operating system with higher accuracy

dominotw 1 hour ago |

Netflix must have the most overengineered tech stack out there. Ppl hired with such high cash comp must feel the obligation to live upto that comp by creating these sort monsters.

relevant hn

https://news.ycombinator.com/item?id=49878857 - A Staff Engineer's Guide to Inventing Work

Tech industry is overstaffed to the gills . No one really knows how to create value for customers( only for themselves through these overegineered slop for career advancement) . Tech jobs are really sad place to be now.

antonvs 1 hour ago |

> This agreement has a name: an ontology. Ontology is a formal specification of types, the properties they carry, and the relationships they may have. Its unit is the triple, one fact expressed as subject, predicate, object.

Er... nope.

In the limited context of OWL and RDF, that's what an ontology is. It has a much larger meaning outside of that context. As M-W puts it, ontology is "the branch of metaphysics concerned with the nature and relations of being."

Projected into computer science, an ontology is a formal model of concepts in a domain, their properties and relationships, along with axioms or constraints that apply to those concepts.

The "triple" described in the quote is an implementation detail, how a particular formalism (OWL and RDF) chooses to represent a rather pale shadow of what an ontology is conceptually.