Ask HN: How Do You Utilize Your Personal Knowledge Base?

39 points by FemmeAndroid 2 years ago | 33 comments

I am always intrigued to read about how individuals construct, organize, maintain, and expand their personal knowledge bases. There's a wide range of tools at our disposal, including Obsidian, Notion, Markdown and plaintext, Zotero, Evernote, among others. Likewise, organizational structures are just as diverse - anyone here a fan of Zettelkasten?

Despite a couple of serious attempts, committing to the process for a few months each time, I've rarely found myself referring back to the stored information. More importantly, the process of adding to and maintaining the knowledge base felt more burdensome than beneficial. I've tried adjusting the content and format of what I'm storing in an attempt to make the experience more manageable, but to no avail.

My main motivation for exploring this further is my growing concern about my recollection ability as I age, and I'd like to fully exploit the available tools to counteract this.

So, for those of you who do have a functioning system that you use regularly - regardless of its simplicity or complexity - how do you employ it to retrieve information or enhance your understanding of topics? Insights into how your system operates would be useful, but are secondary for me at this point. I feel like I've read about all the possible configurations a system could have, yet I'm skeptical about the tangible benefits they can provide. I would be delighted to be proven wrong.

VoodooJuJu 2 years ago |

I used to be deep into the personal knowledge management tar pit. Most of it was wasted time. Here are the few things in it that I actually regularly use and get something out of. These are just simple zim-wiki text files that I update and skim through a couple times a month:

- Idea list: Ideas for art or other projects that I'd like to try some day. I'm occasionally adding new ones and I've started work on old ones I've added that finally piqued my interest.

- Gift idea list: Things friends and family may hint at wanting or outright say they want in casual conversation. I add stuff to this list then buy it for birthdays, Christmas, etc. Can get some real nice surprises for people when they only mention things in passing and you happen to catch it.

- Book list: Just a wish list of books. Occasionally I'll buy a few to add to my personal library and cross them off the list.

Just understand that most of the personal knowledge management stuff is a waste of time. It's a bunch of feel-good busywork that's typically sold to you first as a cheap idea, then sold to you as products like Roam Research, Obsidian, etc. Can't really beat simple lists in text files or on paper. I even like physical notebooks that you go through and keep a hand-written index in the back of. Forces you to think and revisit stuff.

ReallyOldLurker 2 years ago | |

I use the nearly the same.

- Zim-Wiki for items or procedures that I refer to or repeat often. Zim-Wiki is mostly plain text and transportable, but handles embedding small screenshots. I keep another Zim-Wiki notebook for small projects or investigations. - A simple text file list for in progress items or projects that I am working that have more or less an end date or specific goal - A simple text file list for to-do items, vague plans, and links to check. This file changes almost daily. - A simple text file list for wish list items and purchase ideas > syncthing on all my devices to keep all them all current

In my paid work I have a few additional things, but not much more. The most important is a separate subdirectory for each project's files. Structure it so that it is ready to use in a retrospective repository when finished. Refer back to it often when planning similar projects.

I used to "hoard" knowledge, had a large library, sets of sorted bookmarks files, etc., but I'm finding that I need less and less of that. I think the biggest change took place when I made a short project once with the goal of making my home computing portable and self-hosting. That really simplified a lot.

A big help is being consistent in a format and structure that makes sense to you regardless of the tools you migrate through.

jjice 2 years ago | |

Yeah I have to agree. I really liked the idea of having some sort of common place book of files on my machine, but I never looked at them. I have a few todo/list files like you mentioned and aside from that, I dump old ticket and historical data from my day to day into dated files in case I need them. I rarely do, but it's easy to dump since they're fairly raw.

PartiallyTyped 2 years ago |

I use chatblade, Obsidian, and git. I generate a diff, send it to GPT-4 which gives me a short description of what I changed and then I push to github. This makes maintenance easier.

You can do it automatigally, but that can get expensive rather quickly.

I use it as the means to understand things. I write as if I am explaining things to somebody else, that else is the future me, but in the process you can quickly understand what is not clear in your head, break it down to first principles, and build it back up again.

N=1, but writing has had an enormous impact on my growth.

swah 2 years ago | |

So you compare your explanation of a code change to GPT's?

Or you send your explanation to ChatGPT to rate?

PartiallyTyped 2 years ago | | |

I am using GPT to write the commit message for me of my changes.

Since this is purely text, it summarises the changes / corrections.

JohnFen 2 years ago |

I keep everything on a wiki I run on a home server.

Over the decades, I think I've tried just about every method of maintaining my knowledge base, project notes, to-do lists, etc. And the only one that works even halfway decently for me is a wiki.

It's searchable, the simplicity of markup means I won't get side-tracked into wasting time on layout and eye candy, it's easy to search, if I have internet access then I can access it from anywhere, and the wiki I use allows simple embedded spreadsheets and other such functionality for more complex data.

Plus, it's very easy to allow others secure access to it for when I want to share notes, or keep notes that are updated by multiple people (such as when collaborating on a project), or to distribute very large files.

aditsawant 2 years ago | |

How would I go about starting a home wiki of my own?

wgj 2 years ago |

I started using Foam[0] a few years ago, but the more I used it, the more I dropped all the tedious bits, and it became nothing more than a big, evolving markdown repo.

I switched from vscode (back) to vim, and it has worked as well or better than it did before. I follow my own rules. I like the Zettelkasten idea of one idea per card, but if I put more related things in the same .md file, that's OK. I didn't like the flat directory structure, and so I have dirs organized by category. My /bar directory is inside my /cooking directory, and for whatever reason, that makes sense to me. Ripgrep doesn't care, and I always find what I'm looking for.

This markdown hierarchy, that still lives in a repo called "foam", has become indispensable to me.

[0] https://github.com/foambubble/foam

kingkongjaffa 2 years ago |

The secret sauce is orgmode in emacs.

My personal notes are a .org file that is backed up on a private GitHub repo.

I have sprawled about 5k lines at this point since around 2020.

Some things get a org-bullet (*) and a TODO.

Then I just do org-agenda and it pulls up a nice view of all my TODO items for pruning and prioritising.

Having an automatic agenda and the fact the bullets are tab-collapsible and can be used as an outliner is the only way my notes are not out of control.

My other writing is in apple notes (short lived) or exists in tickets. I’ve tried various things like zettlekasten and obsidian but nothing sticks for me.

My other daily task setting and personal planning is a bullet journal with a future log and monthly logs. Bullet journaling is how to get it all out and process and plan personal things. Actual Knowledge is in the org-mode.

For email I follow getting things done and have an inbox an @action and a @waiting folder. I keep inbox empty and the others get pruned every day.

tonyarkles 2 years ago | |

Lol...

    % wc -l `find . -name \*.org`
    ...
    361427 total

precompute 2 years ago | | |

Lol...

    $ wc `fdfind -t f -e org`
    1839279 16220106 107034844

(lines words chars)

Yes, it's structured.

dmwood 2 years ago |

Funny how often the word recollect has come up here, without mentioning the fantastic tool recoll, updated frequently and available for many operating systems. I use it on macOS for an immense library of scientific and other documents. I think there's a standalone installer, and it's also available via macports. It may be found at

https://www.lesbonscomptes.com/recoll/pages/index-recoll.htm...

It allows me to index and find words and phrases in files of many types AND view their contents independently of external readers, or using system readers, including text, PDF, or word processing files of common sorts.

cborenstein 2 years ago |

Thanks for sharing --

> More importantly, the process of adding to and maintaining the knowledge base felt more burdensome than beneficial

I wonder if being able to recollect things later in an organized way may not be worth the effort for you.

Something I've found is that I really don't need a robust personal knowledge base. My goal is to do my work well (and taking notes to support this is definitely valuable). Building a personal knowledge base doesn't support this goal and has a more unclear value-prop, so it's a drag to keep up with.

Wrote an opinion piece on this https://www.stashpad.com/blog/you-probably-dont-need-a-knowl...

karmakaze 2 years ago | |

Right, I've always found the value in organizing and writing notes is that it forms better memories from which to recollect.

Most of my writing (other than a few references or short-term scratch pads) is "write once, read never".

Pretty soon if not already, we should be able to write in any written form and have a language model search it for us using fuzzy search terms, as well as make inferences based on the text.

creer 2 years ago |

My motivation was multi-sided and I went with a completely ad-hoc structure. I store different kinds of information in very different ways. I sometimes reorganize a section. I don't use any monolithic or gate-keeping tool - the information and data is far more important than the tools. The oldest sections go back about 30 years. I have a few conclusions:

(1) It takes time to reach critical mass. Until then, it's overhead. But even then some content was useful from the very start.(2) After that it became very reliable - it's there and I rely on it. (3) For me, it's clearly not just knowledge base, it's PIM - personal information management - KB is just mixed with the rest. (4) After decades, retrieval is still a mix of ideas and tools - not great. (5) I still wish for more efficient data entry / collection.

In balance, for sure a majority of the content is never accessed.That's fine. The system allows un-foreseen uses to happen commmonly and I like that. Do the positives outweigh the costs over the years? I have no idea. I add stuff constantly, some things I rely on again and again, some query ideas are one offs... and usually they work which is great. I know I have a very solid base for recollection. It's not clear how I would do a cost/benefit analysis.

I feel perfectly fine with the ad-hoc collection and storage strategy. That has worked well enough.

rifty 2 years ago |

I use Obsidian to manage personal knowledge about apps, OS features, and workflows. Some of it has to do with personal configurations; some of it has to do with undocumented behaviour I found out myself, issues, or coalescing fragmented information from around the web. I'm pretty careful to not simply rewrite what the web already documents well or is easily found in a single search. So it is a mix of official documentation links, written knowledge, and personal experiences like the search query paths I used.

The approach is basically "I write what I've used if I still have around what caused me to need it. I associate what I use with where I access the context of the cause". Because issues are context dependant you organize by the context that it came from. If you name a file as the app name you can have the document come up when you search for the app name. Because the content is based on situations and issues, returning to it is predictably probable. I find this is a pretty low mental burden to writing, maintaining, discovering, and navigating. You don't really have to make decisions about where to place something, how to name it, or how to find it because it follows from naturally arising circumstances and contexts.

I don't think I technically need to use a PKMS as I use the special features lightly but it is nice to have them there when you really want them and to have an interface dedicated and designed to deal with servicing the whole workflow. I also like to write down my stream of thoughts too but I keep them separate and use a separate system for interfacing and managing them.

lnwlebjel 2 years ago |

Use of this type of thing has been absolutely essential to my work as a research scientist, as the modern equivalent of a 'commonplace book' (https://en.wikipedia.org/wiki/Commonplace_book). Search is the primary advantage over paper notebooks and so search is arguably the most important feature of any of these 'knowledge bases'. Tagging or other forms of organizing being subsets of search, and in my opinion essential features.

So, how do I employ it? One is to search on a subject I'm interested in or working on, ie to find notes, previous readings and writings, then add to these or go from there. Another is to keep various kinds of lists, (eg lists of ideas, things to read, etc). Otherwise the way your able to link things together with tags and other forms of linking is also pretty great - these are all features I will insist on when I move to something new, since I use tiddlywiki, which is a mostly dead project.

Anything with these features works like an external brain, allowing for recall, with just enough inefficient that you have to engage with the material, and possible mix some notes and facts together to hopefully inspire some new ideas.

znpy 2 years ago |

I run a mediawiki instance at home (a few containers managed with docker compose) and it's been great for the last 3+ years.

There are a number of feature for organizing content (the main ones being Categories and Namespaces) and it has APIs (so i can write simple python scripts to perform actions -- like updating some index pages with a custom format).

The VisualEditor is very nice and it's fairly easy to embed images, videos or even pdf documents.

It's not exposed publicly but I can access it (both for viewing and editing) from anywhere using my own vpn connection.

I update the main software once in like 6-8 months so it's fairly low maintenance (this boils down to essentially changing the version tag in docker compose file, restarting and checking everything's working alright -- with the occasional fixes).

Since it's running at home (on a small computer that i would keep on anyway and that it's also doing other stuff) it's essentially free and i don't have to worry about space or any other metric.

PaulHoule 2 years ago |

I have a smart RSS reader I am working on called YOShInOn. The main UI looks like TikTok or Stumbleupon, I can thumbs up, thumbs down or favorite articles, out of maybe 2000 articles a day it picks maybe 300 to show me, maybe I really read 200 on the average day.

Once the articles get favorited though there is the problem of navigating them, I just added some simple text search capability but there is a huge amount of room for improvement there. If it gets the ability to ingest articles via bookmarklet than it becomes a bookmark manager, add some ability to enter notes locally and it becomes a more general knowledge base manager.

It could really use "tags" but when it does these will be tags for the ML age, particularly it will be possible to have positive, negative and indeterminate tags. If the ML system thinks a tag applies it goes into indeterminate status and could be set to negative or positive to make it a training example.

samsquire 2 years ago |

My unit of information is "the idea" and they all go into a single markdown file with a Markdown heading. The ideas are numbered and I use internal page GitHub anchor links to link between headings.

I had a WikidPad Wiki (python desktop based wiki software for Windows and Linux) that has 1000 entries. There was a number of pages that were long. But wikilinks in that wiki were often definitional (rather than explain inline, they had their own page) and multiple pages linked to them. The homepage of the wiki is a launching pad for links to other pages.

That wiki is a design of my ideal computer system, it's on GitHub as idea-wiki.

I think I prefer the all in one file thing because I don't need lots of very small files with very little information in. I also read over my notes regularly.

If you want value from your notes I feel you can read over them regularly.

tmaly 2 years ago |

I had a similar issue. I had read How to Take Smart Notes. This discussed the slip box method. I was recording ideas in Obsidian. But I was not reviewing them.

Fast forward another year, I saw a mention of a book called Building a Second Brain by Tiago Forte. I bought it and read it in 3 sittings.

The concept was that you are only capturing what resonates with you. Stuff that is actionable for your projects.

The structure of your folders helps with this. So current projects are easy to locate. You can keep working on them.

He suggests in a bonus chapter on tagging to just use the search mechanism of your app. He gives some other useful ways tags can be used.

Fast forward to today, I am actively reviewing my notes and improving them each time I touch them to make them more useful.

whorleater 2 years ago |

I keep all my zettles at https://wiki.malloc.dog/.

When I was doing my masters in anthropology at the same time as my SWE job, I ran into this issue of having too many notes and not enough organization. Random other tools never quite did it, generic search didn't work, notion et all kind of sucked, so I ended up rolling my own thing on top of Org roam. It's all plaintext org files, easily accessible.

When I do scholarly work, it's great because all of stuff is in one place. It's also useful for tracking the breadcrumbs I do for my real tech job.

jbottoms 2 years ago |

I also collect these constructs. It's an obscure science; one with a founder; C. Wright Mills. I'll honor his work by using his term: "Intellectual Craftsmanship". This is covered in his book, "The Sociological Imagination." It appears as the appendix of the book, and a website can be found at the site below. You will also find in Google results that there are multiple courses of this name in sociological studies. http://www.sociological-imagination.org/

snisarenko 2 years ago |

The only practical use I've found for knowledge base / bookmarks is to have a searchable list of references/docs/cheatsheets/examples that are hard to find via google (or are not available on google,i.e. internal company docs).

With the release of ChatGPT my need for a knowledgebase has decreased by 10x. And I assume that the need for knowledge base/bookmarking solutitions will disaapear completely in the next few years as LLMs improve.

anonzzzies 2 years ago |

I use Joplin and I am playing with emacs orgmode. So far Joplin has been a lot better at finding stuff I forgot about but knew a few letters that would be there (like ssh). I have many 1000s of notes; from emacs to notepad to ms docs to onenote to I forgot to Evernote to vim to notion to Joplin. And maybe back to the emacs. I use and have used it daily ; it makes me think clearer to write things down before implementing or discussing. I basically write down my mind voice to remember it verbatim.

disadvantage 2 years ago |

I use the Standard Notes[0] free tier. I organize notes with tags. I self-actualize things with my TODO, and everyone should do that. I'm often reminded of the phrase: 'Because I said I would'.

[0] https://standardnotes.com/

mikewarot 2 years ago |

I have my photos and videos in a tree structured by yyyy/yyyymmdd/ with directories created by the camera under that. This helps me remember what I was doing and when.

I keep a WikidPad wiki locally. In the past I used word2000 outline mode for logs at work.

epirogov 2 years ago |

take only text notes. folders hierarchy is categories. strictly split paragraph on meaning complete information within a subject. use string similarity vectors to assign tags, not to text lines. use tags metadata with bot.

softwaredoug 2 years ago |

I don't believe in a "personal knowledge base" I believe in blogging, then finding my stuff on google when I have that problem in the future :)

jorisg 2 years ago |

I use Obsidian with several thousand notes for everything "writing" (structured - studying, definitions, thinking things through, and unstructured - memorable moments, stories, people, random streams of thought). "Actionable" things (todo lists and tasks) have their own place. My motivation is quite similar - I feel like I just forget things. And it's been wonderful! It's so nice to write about a nice moment, and be reminded that it happened - small things you'd otherwise forget.

I've found ideas from Andy Matuschak extremely useful - https://notes.andymatuschak.org/About_these_notes?stackedNot...

On a practical level the crucial features of what's been working for me for a few years now (inspired by Obsidian community and Andy):

* Notes have to be very clearly "bounded" - break things into smallest logical "statement" units, and interconnect notes instead of having behemoth notes with everything in one place

* Note titles have to be an explicit "statement or sentence summary of the contents, with relevant context" - which really helps with search / discoverability. E.g. "Browsers are the most common and visually advanced applications that make use of the internet, but there are also many others" instead of "Web browsers" or "Applications that use the internet". Or "The day I embarassed myself at football". An alternative of this is to have explicitly clear headings.

* The idea of putting notes into a "reverse spaced repetition" algorithm where instead of spacing things you've learnt further and further into the future, you are spacing things that are "fruitful" nearer and nearer in the future (and when you have less to add, further into the future, when eventually a note becomes "evergreen"). You then have a "queue" (mine is always packed) of notes that are due, and you pick them up as you like. This can be achieved with https://github.com/st3v3nmw/obsidian-spaced-repetition and is described better by Andy: https://notes.andymatuschak.org/zJrfPCbY7GcpV9asEc8NTVzXTAV4...

With the above things combined, I usually sit down just to write and unload things from my mind and enter notes into the (basic) "attention algorithm". That way I never stress about polishing notes, and there's always a time in the future when I come back to a note written a long time ago, make some edits, maybe split it into a couple of notes, clarify the title, maybe merge a duplicate (if something has come to mind twice, it's clearly important to me!!). With explicit titles I almost immediately know what the note is about, trigger new connections (between the time I've written the note and now, some other relevant thing may have surfaced which changed what I think or know, or I've come up with a clearer way to describe the same thing).