Ask HN: How do you organize your knowledge? Methods and systems for grasping, understanding, and making sense of any general knowledge or information. |
Ask HN: How do you organize your knowledge? Methods and systems for grasping, understanding, and making sense of any general knowledge or information. |
For a solid understanding and remembering, I’m applying the knowledge by setting up a personal side project.
I furthermore push all my side project to a private git repo and if I really need to look up things, I always know on what project I need to look at.
No-cloud (except for git, could be self hosted), local-first, plain data formats solution for note taking, knowledge organisation, text production and spaced repetition.
I use the mac OS/iOS notes app to keep track of short lived information that I don't want to forget.
Tiddlywiki for a public knowledge base
IMO, a prior question to understanding the information we consume is deciding what information to consume. There's an essentially infinite amount of content out there, but we have a finite amount of time to learn. The strategy we usually employ is random and mindless—we consume whatever happens to show up in our HN or Twitter feeds. But I think there's a real value in thoughtfully curating the knowledge we take in, which is why I built Trove [1,2].
I'm curious to hear, how does everyone else here optimize their information diet (if at all)?
I used to take notes at school, but hardly ever read them again, it was the act of taking notes that helped me make sense of stuff and remember it.
Information capture happens in a variety of media and goes into a physical or digital "inbox". I have both kinds because sometimes I take a note with pen and paper, or I have business card, or I have my cellphone and I can take a picture.
My inbox includes the "pictures" and "downloads" folders on all of my devices. I link my devices together using SyncThing.
Then I have an org-mode file I keep my projects, someday-maybe's, and project archives in. My reference "file" is a 1/2Tb drive split into folders. My file organization is pretty flat right now, so at the top level I have "school" and "recipes" and "day trips", etc.
This is an utter cludge, but it does most of what I need it to. I've actually adapted it some to use at work and that has allowed my to stay very organized with work projects.
I personally don't have the discipline to organise my digital notes post-hoc, so I focus on just making notes using markdown files and sticking them somewhere in my onedrive so I can find them later. I do have a notes folder, but there is stuff that has found its way elsewhere. Thankfully searching markdown files is very easy.
I consider it like a tool for clear thinking. There’s no punctuation or noise, pure signal. I build DSLs and then spreadsheets in those DSLs. Both the data and grammars are in 2D Lang’s. There’s no quotes, brackets, patents, etc. Everything is strongly typed (eventually, in practice I start with a lot of “any” fields and iterate on the ontologies as my knowledge of a domain grows).
Everything is backed by git.
I do a lot of visual modeling lately with my remarkable 2 eink tablet.
1. A zettelkasten, using [neuron](https://neuron.zettel.page/), synced with git (and hosted at github pages), as well as another private repo that I self-host on a machine at home
2. A series of dropbox-synced plain text markdown files edited with vim and accessed via a few fuzzy finders - feels just like notational velocity. these files and the way i use them are documented and organized by a single root file, everything spreads tree-like from there.
3. VimWiki (deprecated. migrating into zettelkasten as appropriate)
4. A small moleskine notebook that is always in my pocket.
Knowledge itself I keep written in markdown files. I keep folders, multiple files, and headings within files to organize and group things by topic. Everything is in unordered bullet points. The bullet points can be nested as well.
Indentation from nested bullets, plus the syntax highlighting for headers gives visual indication of where related material is located.
Joplin is opensource and uses Markdown files. Syncs with WebDAV. It's the best I found till now.
LibreOffice is for tables since Markdown sucks for them
There's a way to implement full GTD [0] framework with Evernote using Notebooks and tags. This is a very useful part for me, especially long-term and life planning horizons.
And save all the things I learn into mdbook hosted on my github.
Here’s the link: https://til-mraza007.vercel.app/
Here’s the source: https://github.com/mraza007/til
I put everything into a markdown file and save it on github
It's a more flexible mind map that allows adding urls, videos, images, richtext etc.,
I have different maps for different topics - like "things I want to learn", "my ideas in XYZ", "project A" etc.,
Anki
Bear
Bulletproof Workspace
Commonplace
Dokuwiki
Emacs (org-mode)
Evernote
Everything
Folder structure with files
GTD
GitJournal
Github
Google Docs
Google Keep
Joplin
Markdeep
Markdown
Mental Pivot
Mr. Dewey and his decimal numbering
NeuraCache
Notepad++
Notion
Obsidian
OneNote
Pen & Paper
plain text (awk, sort, grep)
Roam Research
Spaceli
Sublime
SyncThing
Synology
Trello
Trove
Twitter
Typora
Vim
Wiki
ZettelkastenThe application or method of your choice will only organize your knowledge (more correctly, represent it) if you already have done it yourself in your thoughts.
I just use nicely formatted markdown files these days. There's no magic application that'll do it for you.
It syncs my notes to a cloud UI, my phone, laptops etc. Also provides for good search, and the notes are in markdown format so very portable. There are some limitations (no real way to link between notes) but I don't notice them or do the kind of writing which requires such things really.
I am still trying to identify what makes your technical solution unique and better that, say, a Jupyter/Dataframe based system.
1) It seems to me that most of the commands that have to be done in a UI with Google Docs/Excel can now be written down and immidiately resolved in your hybrid solution. That probably opens a path towards diffing/version control. Type checking is very nice, though.
2) The spreadsheet like UI probably makes it easier to edit data, although that is something I would like to play arounnd with.
Also, I am trying to understand the key conceptional differentiators. What do you mean by 2D language?
> I am still trying to identify what makes your technical solution unique and better that, say, a Jupyter/Dataframe based system.
It's a low level symbolic notation. Any symbolic notation is an imperfect way to model reality. Some are better than others. I propose that Tree Notation may be the best such symbolic notation we have ever come up with. I measure "best" here as a function of capabilities and complexity. Tree Notation has all the capabilities of other symbolic notations (you can build a "data structure zoo" and you'll find that every structure has a representation in Tree Notation), without the complexities of extraneous syntax. You simply have words. You have no quotes, no parens, brackets, etc. Words laid out in geometry. Sure, about 1% of languages have done the "indent significance", such as in Python and YAML, but no one (until Tree Notation) figured out that that's all you need. You don't need any other syntax characters. Simply have rows and columns (and using indents for scope) is enough. This "simple as possible" pays dividends. For example, simple new ways to count complexity (https://jellypbc.com/posts/ezmntq-counting-complexity).
As a low level notation, you can't compare something like Jupyter/Dataframes directly to it. That would be like comparing Jupyter/Dataframes to binary notation. However, you can compare Jupyter/Dataframes to a higher level tool built on Tree Notation, such as Ohayo (http://ohayo.computer/). What does Ohayo have that Jupyter doesn't? Well it's a very high level language where you can mix data and code easily. Coding with it is very copy/paste friendly. Whole programs can be put in the URL. There's no server side backend needed—everything runs locally in your browser. Of course, there's no ecosystem or community behind Ohayo yet, but the nice thing about Tree Notation is that simplicity doesn't go out of style. If at some point the resources spring up to develop it further, I think it could be a very popular tool and community.
There's nothing semantic you can do in Tree Notation that you can't do in Jupyter/etc, as for the foreseeable future they all are running on the same hardware. But it's very pleasant to write tooling and parsers for Tree Notation since it's so simple, and as a user you get great things like seamless semantic interop with git, pleasant copy/pasting, url friendly programs, et cetera.
> Also, I am trying to understand the key conceptional differentiators. What do you mean by 2D language?
All existing 10,000+ computer languages I would call "1-D". That is, it is assumed that you have 1 read head that moves linearly along the program generally from beginning to end. Tokenizing happens, white space is stripped, an AST is built, compilations occur. 2-D languages don't have the same assumptions. You could potentially have multiple read heads. Read heads could move not just back and forth, but up and down. Of course, it will be a while before we have 2-D registers, and so for a long while we still have to compile to 1D langs, but there's a future where computing looks a lot different than today.
Another way to think of it is that currently programs are roughly transformed into a ticker tape and read linearly in order. 2D programs don't have to be, and could be read by the computer similar to how a human reads a newspaper—parsing and zooming in and out in potentially random order.
A much simpler more pragmatic definition for now could just be that a certain subset of TSV is a 2D language.
many thanks for the very detailed response to my questions. I wish I could upvote you more than once.
I am looking at your paper/cited literature to understand a bit more about your work. At first, my initial thoughts were that this is very similar to a Lisp derivative. I now see that indeed there is a second axis in place, be it via tabs or cells.
Thank you also for the link to Ohayo.computer. I will check it out, and try to fool around a bit with the 2-D lang concept.
Cheers,
piotrWhen you don’t have any particular emotionally charged goal(s) in mind — no time/knowledge management technique would really make a difference.
But some of them might give you a good illusion of something important going on.
And as soon as you have specific important goal in mind (like preparing to specific exam, getting lawyer license etc.) — almost any approach you feel comfortable with will do the job.
You usually dont need to review possible options in advance, the goal itself gives you hints and guides (if preparing for exam means remembering a lot of scattered facts - you’d probably end up with some anki cards on your own etc).
In the times where an emotionally charged goal isn't driving me forward, adhering to disciplined regimes for time and knowledge management has helped me to prepare and therefore perform better in times there IS a highly motivating goal.
There don't have to be two extremes of X-management (one fuelled by emotion and one dragged down by apathy). Like anything, building a discipline in the slow times will set you up really well for when you need to run hard in the fast times.
All that said, I use Roam Research[0] for my knowledge management now. I consolidate my thoughts and ideas weekly, and aim for evergeen knowledge[1].
[0]: https://roamresearch.com/
[1]: https://notes.andymatuschak.org/z4SDCZQeRo4xFEQ8H4qrSqd68ucp...
Edit: formatting
But i see that its near impossible to build a discipline without particular goal in mind.
By “emotionally charged” i meant “really important for you”, important enough so you even force yourself when you don’t feel excited about the goal.
For me, the key was switching to a totally non-linear but robustly inter-connectable note system, which turned out to be tiddlywiki in my case. Regimented approaches I found created friction for creating a new submission, and switching to just creating a 'tiddler' when something was on my mind (and then being able to choose to reference that content within larger ones later or not) has been one of the most liberating changes I've made in years.
PS: I don't know the other tools (notion, roam,...) PPS: a big downside in Obsidian, in my humble opinion, is publishing that knowledge on the web. Probably a static CMS on top of the .md file generated with Obsidian can do the trick. But it is a tedious step that I never had/wanted to investigate.
I use Notable rather than Obsidian, but it works the same (folder of .md files). My blog builds with Bridgetown, and I wrote a little bit of extra code to handle the wikilink formatting and build up an incoming link list, but it runs pretty cleanly. https://blog.tracefunc.com/notes/
you’ll be up and publishing in no time (for free and automatically to GH pages if you want)
I find that I do some pretty cool stuff and then totally forget how I did it a year or 5 years later, mostly with coding. The open source projects I documented and explained are things I myself google to help me set them up in the future, so a way of codifying my knowledge has been to try to write about it and put that in public.
I set up development environments for myself a lot, so I wrote an article about how I do it https://modfoss.com/creating-my-development-environment.html and then put the code on GitHub as well https://github.com/symkat/modfoss_devel So if I don't do it for a while, I'll have a starting point and me-from-the-past explaining what I did and why.
I wanted to go deeper for my company, so I bought the Notion system called Bulletproof workspace (https://www.notion.vip/bulletproof/) in December after er doing. ton of research, but found it a bit overkill. In retrospect, I think it was just my newness to Notion that made it seem so.
However, once we reached a certain point where we were overwhelmed and completely unorganized. I finally invested the time to learn how to use Notion and Bulletproof Workspace and it was amazing, now I use it for everything from onboarding, to brainstorming, to-do's, etc. It's quite impressive.
I highly recommend checking them out if you're looking to get organized. The flexibility is what really made it work for me since it works for everything. It's nice just to have 1 tool.
I'm a visual learner, so its really easy for me to recall past snapshots (static, not like a video) from my life, or how certain things looked at a point in time. It's not quite an identical memory, but it's not too far off either. Maps are a breeze, and even complex tunnel systems like the PATH in Toronto are easy to navigate.
For learning, it helps if you just throw yourself into it. Take 5 minutes and try to work at the concept. Usually this will turn into 30-60 minutes. I learned Kubernetes and AWS EKS this way.
For problem solving, I'll make some notes for brainstorming, todo lists, and any questions I have. Then when I go for my daily walk (~15km/2hrs), I reflect on the abstractions again (from memory) and try to refactor what doesn't seem perfect.
I do have a mind palace set up, but I don't really use it.
One's knowledge is too valuable and important to entrust it to a particular binary format that can soon no longer be read. Plain text is durable, portable, easy to process using UNIX command line tools, it can be full-text indexed with a reasonable overhead. It can be version controlled easier than binary formats or formats with heavy markup.
I often summarize scientific articles, write down new ideas or need to preserve how I did something (run a system, install a tool) for later replay, and plain text is great for that for the most part. Occasionally, I used LaTeX commands e.g. for embedded $ maths $ or #hastags to tie together files by topic for indexing.
Importantly, my workflow is OS independent (I mostly use Linux and occasionally MacOS X/iOS) and editor agnostic (I use Sublime and Emacs).
I would very much like to hear from others how they address their KM needs.
As I couldn`t find a good tool I wrote my own Wiki server - 13 years ago and still using it every day: https://moasdawiki.net/
It is intended for memorizing things, but the fact that it forces you to divide knowledge into pieces small enough to fit on a flash card also helps to organize it.
And you can have different card stacks in Anki, so there also is categorization of knowledge.
Cf. the "Zettelkasten", a physical flashcard system which is actually intended at organization, not memorization: https://en.wikipedia.org/wiki/Zettelkasten
Anki can be used to emulate that digitally.
"topic: D3",
"subject: scales",
"context: Side Project"
Incredibly flexible, a bash-like shell is my only dependency. Works with any search engine that i've tried. Easily replacable parts. Check into git regularly.
Some scripts I use
## group by topic. search for lines that start with "topic". print topic and the file name
grep -i "^topic" *.txt | awk -F ":" '{printf "%-25s%s\n",$3,$1}' | sort
## find all files containing the "topic" tag
grep -i "^topic" *.txt | awk -F ":" '{printf "%-25s%s\n",$3,$1}' | sort
## find all files NOT containing "topic". useful for cleaning up
grep -iL "^topic" *.txt
## find first 10 files not containing "topic" and open each in vi sequentially
for f in $(grep -ilL "^topic" *.txt | head); do vi $f; doneWould you have a repository of your scripts somewhere?
It looks like this [2] (#000000 black interface for mobile due to OLED display).
It changes context depending on time of day and device used. Eg. before working showing me the info I need about the kids [3].
[1] https://writings.stephenwolfram.com/2019/02/seeking-the-prod...
[2] https://files.littlebird.com.au/Shared-Image-2021-04-26-07-5...
[3] https://files.littlebird.com.au/Shared-Image-2021-04-26-08-0...
Digital notes are good for a lot of people and a lot of situations but not for my long term very general knowledge based tasks. For specific commands/documentation/syntax/techniques yeah digital notes are nice. For Brain storming, and connecting ideas though? I fully believe paper works best. It's more flexible.
I like your idea - what ever happened to that e-ink note pad one could write on and it saves a digital copy?
Probably https://remarkable.com/
I can always revisit the notes to quickly refresh myself on the topic (if a context-switch happens later down the line, or if the topic is not related to my day2day work).
I've successfully used this approach to onboard myself for a Spring project (known to have a steep learning curve). It was my second time using Spring, with an 8 month time separation. The ramp-up/refresh took about a week.
There were also some positive unintentional side-effects of my approach. I keep my gitbook notes public. I found that making my notes public ended up helping others on other teams in my company. Specifically, more experienced Spring devs who were new to Webflux were able to use my notes to jump right into a client's code base.
Something about like... a syntax similar to Markdown that is [[deep-linkable]] or something that allows you to... create really deeply nested queries (almost like a GraphQL vibe) to finds things that are "loosely/tangentially related"
Something like... if you were to go to Wikipedia and play that game where you start on one page (a kind of bird) and try to get to a different page that is 50-60 deep/nested links away (a World War II event or something).
I personally am not a big note taker, but boy oh boy do some seem obsessed with special note-taking syntaxes/apps.
1. Commonplace Book This is a book of thoughts, findings and general collections of information captured throughout your life and brought together into one place. [How And Why To Keep A “Commonplace Book” | Thought Catalog](https://thoughtcatalog.com/ryan-holiday/2013/08/how-and-why-...) [Commonplace Books Part 3: Choosing a System - GeekDad](https://geekdad.com/2020/03/commonplace-books-part-3-choosin...)
2. Zettelkasten
A different concept, but also with relevance to storing information for a lifetime. From my brief readings, there’s more emphasis placed on two stages of note. The first is a quick jot - an addition. The second is a more thought out, self-contained existence of the note. There’s also focus placed here on _linking_ notes together.
A good example of someone’s Zettelkasten is [[Andy Matuschak’s Note Collection]]: https://notes.andymatuschak.org/About_these_notes?stackedNot...
[Zettelkasten - Wikipedia](https://en.wikipedia.org/wiki/Zettelkasten) [Bear App: A Solid Zettelkasten Solution for iOS Users — Mental Pivot](https://mentalpivot.com/bear-app-a-solid-zettelkasten-soluti...)
Writing helps structure and encode information in my mind
If I forget it, it’s on google, and I often land back at my old posts that I reread.
It might also help other people, but really future me (and possibly close colleagues) are the primary audience.
My main goal is to have everything in one place: My journal entries, minutes from meetings at my job, project plans and working notes, book/article/film summaries, quotes, reflections. (Bear has both a Mac and iOS app which means I can note things down from anywhere.)
This enables a few things. Search across all my information from one search box. A strong incentive to take good notes because I know I'll be able to find them and reference them later. A 15 year chronicle of the things I've done.
The older blog posts I've written aren't in Bear. This is kind of dumb because they're some of the deepest and most hard-won knowledge I have. Long term, I want to move them into Bear. More recently, I started a new blog that I publish from Bear. This feels much better.
As for learning, I have a few processes that relate to recording:
1. Highlighting useful/interesting passages in books and writing notes in the margins with my thoughts on the material. For really good books, I'll pull out the highlighted passages and notes and organize them into a summary of the book in a note in Bear.
2. I have a note for each skill I'm working on (e.g. designing software architecture, estimating time frames, getting buy-in on an idea). As I practice it I'll write down things I've figured out or reflections on my application of the skill or relevant notes from books/articles.
3. I'll sometimes reference my notes about books/articles as I'm working. E.g. For some reason I've referenced the note that contains a summary of an article I read about the React lifecycle like a zillion times.
The information collection: When I'm learning new things and sorting by what is interesting I simple bookmark a lot of stuff. It could be a blog post, a research paper, a book recommendation etc. All this is raw information until I take the time to consume it and digest it. Here I use https://lxi.ai/ (disclaimer: I built this) to keep my bookmarks organized with a lil ML.
The knowledge keeping: While I consume/digest I take notes in obsidian. I like working in markdown and having everything stored locally is something that keeps me comfortable. The real key here is putting in the time and actually merging what I am learning with what I already have in my obsidian "second brain".
What I would love is a way to index & search ALL my digital assets: audio, video, images, docs, urls, code and binaries
And it would help if I could use a query syntax to create custom time ranges
Recently noticed Weezer frontman Rivers Cuomo taught himself Python via Harvard's CS50 to organize his music demos dating back decades!
And I've come to realize virtually every one could use such a personal digital archive interface ;)
https://techcrunch.com/2020/11/25/rock-star-programmer-river...
For memorizing things, I use Anki which looks old but it's great. Every day, I spend a few minutes on it reviewing things I learned before and wanted to engrave it in my memory.
I'd suggest an interesting related interview (not mine): How to Make Yourself Into a Learning Machine (https://every.to/superorganizers/how-to-build-a-learning-mac...)
https://en.wikipedia.org/wiki/Wikipedia:Contents/Categories
Once you've picked that scaffold to suit your needs, then no more than two or three indices before a piece of info (depending on how many subcategories you'll want to labor on) is easily searchable.
Also on a side note, we must all be wary of epistemophilia!
Then I use this Custom New Tab URL extension to turn my new tab page to Spaceli: https://chrome.google.com/webstore/detail/custom-new-tab-url...
Edits: It is my "self-program" for life as I learn, make plans, tasks, calendar, study topics, work on habits, map out details of things, track contacts, etc, etc. Progress has slowed due to health limitations, but I still have many plans. It is text- and keyboard-oriented, feels today like "endless lists of fast recursively nested lists", very physically efficient, can export to html or numbered or indented text outlines, import (somewhat) from text files, show an activity log by date (~"journal"), can store files (not slick, but works), etc.
It is, today, a .jar and you have to install a JRE and postgresql and follow documented config steps. For me at least, the best thing for this that i have found. Code in github.
Another future feature is like sharing, so it becomes like a (selectively) public wiki+gopher server, with updates selectively synchronizable across instances. Down the road, hopefully. And embedded code associated with entry types.
Like I started with Notepad amd later switched to Evernote in 2009 afterwards to OneNote.
I used to save links so that can referback as well. I wrote a small app, so that I can save small notes and links https://stash.bobbydreamer.com it uses firebase. Last year found out saving links is useless when the website upgrades or they possible close down. So I need to take a screenshot or copy that info back to notepad or some other means.
Now im back to saving in onenote, I have an half built website which I got inspired as next version of stash.bobbydreamer.com using firestore and writing content in markdown and viewing it as a HTML site. I liked I this idea but really do I want to do it is the thought.
Last year during lockdown I had rethought that should post my notes in organized way in my site so that I can referback easily ( https://www.bobbydreamer.com ), site uses Gatsby, so the notes are in Markdown, don't want to focus on design at the moment as working on lots of things, I never get anything done. There are so much notes I have and to make something complete, it takes time as ideas, notes and thoughts are not always complete.
Theres not much traffic to my site even being public, so I need not worry about it.
I have made my mind that it will be strictly off office work alone(study notes).
Currently OneNote and Personal site.
I have a practice of creating a directory `Machine_<host_name>` for each physical device. And, further I name each file `<host_name>-*`.
But did I name the file LUKS? LVM? or name it after my distro? or just partitioning? (of course, in truth, I can search in text files for those same keywords, but really I was just overwhelmed by the disorder of my folder)
So I was thinking about this problem. Now a bunch of odd folder names just doesn't communicate in a pinch. Now I'm thinking along the same lines of how programs are packaged. Specially named folders. Structures of folders.
I'm thinking along the lines of:
+- Machine-<host_name>
|- issues.d
|- docs.d
|- configs.d
|- fixes.d