Ask HN: How to organize personal knowledge? In the last few years I have collected a large amount of ebooks, papers, articles, movies, pictures, etc. (all digital). I am at a point where I would like to organize these items into a cohesive, searchable and maintainable system. I have looked at various methods (e.g. Niklas Luhmanns "Zettelkasten", Zotero, Org-Mode) and I am not satisfied yet. Any tips? |
- Evernote - It's been a while, but at the time it had no highlighting. Formatting text was also a bit of a pain. Attaching large files was a problem because of free limits (I didn't care about sync but you need it for clipping). But the biggest problem was organization. The notebooks were not enough and it was really slow having everything in one account. Tags did not really help the organization problem either because what I really needed were custom fields. Search was slow and it was hard to be precise because there's no custom fields to be precise about. Additionally for any actual writing, there's no note history on the free plan.
- Most of the Evernote alternatives - Although I didn't move my collection to them, I did try every alternative I could find. If there's any in specific you're considering, I have probably tried it. The problem with most is they were either too simple, buggy, old/unmaintained, or just plain Evernote clones, which already wasn't working. Also many good ones that might have worked were subscription only because they were cloud solutions with sync. Then some sounded promising on paper (wiki-like alternatives), but they would just make me waste time linking and weren't really what I needed. Also many use only tree organization which just does not work for me. Or they don't support attachments well. TagSpaces looked really promising for an on-disk solution but didn't cope well with moved files when I tried it.
- Scrivener - This is what I ended up moving to. It's not the most intuitive piece of software. Development, especially for the Windows version is incredibly slow. It used to not have sync, now it sort of does, but it's not like Evernote (no clipping). I don't really care for sync though, also I like that it was a buy once sort of thing. Now it's still pretty slow with search (made worse by the fact it sort of freezes instead of showing some loader), but I can divide what used to be my main Evernote notebooks into completely separate projects. You can also have custom "views" (lists of notes) which helps reduce a lot of searching. You can also easily search just one specific field although I do kind of miss Evernote's query language. What really sold me though was you can have a completely custom organization scheme. You can add different types of labels/tags, you can change the note icon, you can color code notes, but best of all you can set custom fields (it's like a really fancy database really). You can link to other notes. You can have a custom note card summary for the cards view. There's several different view types (list, same tree-depth together, note, card/corkboard). It was designed for writers, so it's easy to write in and keep everything in the same format, etc (all copied articles are nicely formatted about the same). You can take as many history snapshots as you like. Best of all you can have huge attachments. I have videos, pdfs, etc (known image formats and pdfs, you can see inside the program, videos, and unknown files open externally). You can also set a custom theme and shortcuts which was also really important for me (Evernote still doesn't have a dark mode!). Also although the on disk project structure is not exactly easy to navigate, it exists (i.e. attachments just get copied to a folder in the project, all notes are plain .rtf files) so you can use Drive/Dropbox to sync to other computers or edit notes externally.
I'm working on my own custom solution now, but Scrivener has worked the best for me for notes, articles, and "clippings" if you have a lot of that. I would not used it though for ebooks, or movies. Ebooks I use Calibre because it also has the tools to edit them, though you could have ebooks in Scrivener if you want, and Calibre as the reader. Movies I have only a few, and just use a folder. For quick notes I use Google Keep or Simplenotes. Then every once in a while will clean them and extract the good ideas to Scrivener.
Also, throw everything into your ~/Downloads folder and use ripgrep.
Im not kidding. For any task you dont need more than 5 references. Dont keep useless volumes of stuff and move on.
Disk indexing isn't enough to answer "where is that file", when you moved it for example to cold-storage.
Note that by this definition Windows is not an OS :) Linux came closest with the failed Nepomuk.
Simplest method there is is the file system and files. That method is offered on each computer. Make your own hierarchy.
In general, I suggest 2 folders, one for individuals, one for groups.
In the folder of individuals, you are sorting by name or also by their ID number their files, files that arrived from individuals.
You may symlink the individual folder to a group file, thus knowing which individual belongs to which group, or individual may be symlinked to multiple groups.
Within groups, you can have further hierarchy, those can be groups of knowledge, for example "All about God" put in one folder with the hierarchy below, there you can place your research on Elohim, Summerians, or Annunaki, whatever you may want.
Other folder may be for your personal mind development, you place things underneath.
Some folders may be related to the humanity, groups, countries, and similar.
Sciences can be organized that way.
All you need is a computer, text editor, and file system, and knowing how to use the file system.
People placing everything on "Desktop" have yet a long learning period to go to understand why is there a file system.
[0] http://www.brettterpstra.com/projects/nvalt
[1] http://1writerapp.com
[2] https://github.com/jrblevin/deftFirst, that helps me to rephrase knowledge in my own words (except for quotes, of course), which is good for retaining knowledge.
Second, I use tags and note links that help me to navigate through pieces of knowledge.
Third, I only store files in Dropbox, with a very well-maintained folder structure.
I don't clip stuff in Evernote and don't just copy/paste. I think that bloats the whole thing up and makes it unnavigable.
"sitemap.html" and "catalog.json" are the most relevant parts. The project itself assumes you already follow some organizing principle based on the filesystem, and documents it for you, with links.
It's based on Makefile, and existing utilities, so you should be able to use it as is, customize it, or as a start of something better.
for a universal knowledge/media/file organizer the way might be a smart tag-based system, could not find one still.
used drupal, wordpress for CMS in the past before switched to gitea. Can't use static-site-generator as I don't want to put random personal notes on internet without even a login.
a simple, login-included, markdown driven cms with search/tagging will be really nice.
I found and paid for https://www.buildingasecondbrain.com/ . It was immensely useful for me. The course provides a workflow specifically on managing all the information that feeds into our lives and shows you how to use Evernote to do this. Happy to answer any questions from my experience.
I then periodically clump similar content into a single Keep note, usually by simply cut/pasting the relevant links. This happens once every 3-4 weeks.
Now if only Google's Keep team would publish an API...
For notes: First condense the thought and put it nicely to Dynalist - Because it is all easily searchable, linked and think-then-write tree-structure. - Mobile app is a must to capture ideas, thoughts on mobile.
For todo: Things/Todoist etc. would work. - Because you need deadlines, reminders.
An idea on my back burner is finding a solution for auto-generating taxonomies using Python NLP. But I've been saying that for years. Soon.
I keep a lab paper notebook for everything I do and I write the date, time, and whatever I need. Then I type it in.
The idea is that you can publish/edit wiki pages, "twitter-like" messages, pics, videos, do whatsapp-like group messaging etc while maintaining the privacy of what you write.
Wiki: I host my own mediawiki instance.
Documents: I host my own upspin: upspin.io backed by ZFS.
Images: Combination of upspin (above) and iCloud Photos.
I also export final documents there, eg Mindnode maps, Presentations etc. I write notes using iA Writer.
iCloud: Documents, Photos, Videos, Music (Production), Programming
Omnifocus: Reminders, Project Tasks
I organize everything important in dynalist or file folders in dropbox
Whats importanf for me is I can take my notes from any solution and not rely on any one service
Yes I'd be interested in having a tool to keep videos organized.
The biggest disappointment with Notion is that there is no on-prem. I would've rammed it through at work by now if there were. I've searched high and low for a comparable open-source alternative, and the best I get is outline: https://getoutline.com, but it's got a long way left to go.
It gives you just enough context preview around your search term to judge if the PDF is useful or not. Currently using with 10k+ research papers and very happy with it.
Trello is absolutely great for this
I have several more or less organized ones for specific topics and a catch all called "personal knowledge base"
with cards for a bunch of random things there is no more specific place for but not enough for it's own board
And I use Gingkoapp to organize my thoughts.
http://vincentmtang.com/toolbox/
I only put all the ways I organize my things via software, still a WIP (work in progress) for hardware things I use (pens, notebooks, etc)
------------------------------------------------------------------
## Notetaking / Productivity
Dynalist – Primary notetaking software
Airtable – Google spreadsheet meets microsoft access. A swiss army knife for business. Very useful for making a minimal viable prototype
Pinboard – Bookmarking. I consider this 3rd party chrome extension a must-have, I bind it to my CTRL+D key
Anki – Flashcard software. I use it to speed up learning math, programming languages, and computer science.
------------------------------------------------------------------
## Software Development
Chrome – My everyday browser
Slimjet – A simple chromium based browser that I use for webdevelopment occasionally. Because then I can have a seperate desktop icons
VS Code – My primary IDE
Github – Git + cloud storage for programming projects
Codepen – Isolated frontend environment. Used to A/B test components and explore cool things. Organize codesnippets here like I do in evernote.
Postman – API testing
LucidChart – Making flowchart diagrams and data models. Not free though, for that I prefer draw.io
------------------------------------------------------------------
## Design Tools – (macOS and windows)
Flux – Makes screen red at night so I can sleep easier
AffinityDesigner – I use all adobe products as well, but I prefer this over illustrator
AffinityPhoto – Same reason, I prefer over adobe photoshop
AdobeIndesign – Catalog & Brochure design
Figma – UX design tool, applesketch is macOS only
AutoCAD – Architectural drafting program, but not free
Fusion360 – 3D modelling program for DIY projects
GooglePhotos – Photo & video backup
Dropbox – Cloud storage for files/folders
------------------------------------------------------------------
## WINDOWS SPECIFIC
Phraseexpress – Keyboard macro automation. Checkout my 14 phrase-express guide to see use cases
Autohotkey – When I need an advanced macro that phrase express doesn’t work on. It has a high learning curve and ugly syntax language though
DirectoryOpus – Native windows file explorer sucks, this is best file explorer in market. Has a learning curve. Checkout use cases here
7+ Taskbar Tweaker – Smaller icons on my windows desktop screen
Blank Spacer Exes – It gets hard to look at so many icons in windows, so this gives me eye relief by grouping icons together
Greenshot – The best windows quick image editor out there. First thing I install on friends/family PC normally, its that good. MacOS version is not good.
ShareX – ShareX is greenshot image editor that supports gif uploads / custom image-hosting options. Version 11.6 is the best. MacOS has no equivalent.
------------------------------------------------------------------
## MacOS specific – alot of programming apps
Iterm2 – A better terminal environment. I binded it to ⌃⌥[spacebar]
Oh-my-zsh – My preferred bash alternative for development
Penc – Gesture based window snapping/resizing
Spectacle – Keyboard based window snapping/resizing
Keyboard Maestro – Keyboard automation for macOS. Similar to phraseexpress but better
Alfred – Better alternative to macOS’s spotlight for quickly opening files / apps. I binded it to ⌥[spacebar]
Captured – An inferior version to shareX without gif support, but best thing I could find.
Dropzone 3 – For quickly handling common tasks like moving folders around
Cheatsheet – Because I’m a new mac user I need to quickly memorize commands
Unarchiver – So I can one click install things on macOS
------------------------------------------------------------------
## Chrome Extensions – Productivity
Silverbird – For quickly tweeting things on twitter
Tabsnooze – Set reminders to myself to pay bills at end of month, and to do incremental reading
Pagenotes – When I need to save some specific information on a website. E.G. terminal commands, regular expressions on regex101.com, etc.
TheGreatSuspender – Suspends unused tabs after 30 minutes. Because chrome is a memory hog
Tampermonkey – Injecting javascript on a page to enhance notetaking tools. Its like your own chrome extension. Read about how I use it here
Ankitab – Relatively unknown, but its the opposite of Momentum. I use it to review flashcards thoughout the day
DynalistCompanionClipper – I use this with my notetaking app dynalist, I dump all my ideas here as they happen.
Stylus – Add custom CSS to webpages. Since stylish was tracking user data and got banned
VideoSpeedController – So I can play youtube videos up to 16x speed with captions to speedread. Its how I consume information very quickly
Imagus – Hover over an image to see its full view. Useful for my notetaking app dynalist, as well
GoogleDictionary – Select a word to see its definition.
PowerThesaurus – Select a word to see synonym and antonyms. Thesaurus are my favorite business and coding tool, naming is important
ClipboardHistory2 – SelectAll, CopyAll – you now have a saved copy of whatever your typing on reddit / hackernews / wordpress
Lastpass – Chrome extension for managing passwords. Cross platform
AmazonAssistant – So I can dump all my favorited wishlist items anywhere to amazon, export it out later
------------------------------------------------------------------
## Chrome Extensions – WebDevelopment
Wappylzer / Builtwith – To check what websites are running these days, accurate 50% of time
Page Ruler – Measure things on screen quickly
Webdeveloper – Powerful swiss army knife for webdevelopment, but I never use it
ViewportResizer – For responsive webdevelopment, I use this daily. Better than chrome’s native tools. Checkout bookmark version here
Gifscrubber – I need to pause gifs especially if I want to make a codepen of it from dribbble, etc. PlaytheGif is a close equivalent.
RefinedGithub – Better looking github made by Sindresorhus
Octotree – Left side file explorer for github repos
------------------------------------------------------------------
## Android / General
ColorNotes – Every other noteapp I’ve used sucks. I write things here, copy+paste it to dynalist later
GoogleKeep – When I need to quickly sync an image or text from my phone to PC and nothing else works
Meetup – Use the specific link, login, and add URL to homepage. Meetup android app sucks. Shows only the 28 techmeetups I’m subbed to
SolidExplorer – File explorer
Should I answer? – Tells me if phonecaller is a scammer. Not that I pick up phonecalls I don’t recognize anyways, that’s what voicemails for.
Slack – Team communication
OfficeLens – For making PDF’s from documents using your phone camera.
waze – If I know a driving route is slow, I use waze
okgoogle – My favorite commands are: “Play spotify”, “Next Song”, “Navigate Home”, “Exit Navigation”, “Find Parking”
Spotify – Music
Musixmatch – Lyrics for spotify
GoogleCalendar – Calendar
------------------------------------------------------------------
## Content / News
Youtube – I only put an alert on people that post high quality content infrequently. E.G., SmarterEveryDay, MarkRober, TomScott
ProductHunt – Where I like to see all the new fun tools out there
Hackernews – Lots of smart and interesting people here. Also, where I find out about news in the tech environment
Reddit/subreddits – Copy this multihub, delete ones that aren’t relevant to you, add cool ones like r/welding, r/cableporn, r/engineeringPorn
it supports attachments and other stuff
eg when i research “how to do password less ssh”, i will create a note with that title for future use. it has worked so far.
devonthink if you want a personal knowledge database with good searching
A caveat is that files are limited to 250Gb currently and private repos are 100Gb personal and 100gb per team. I'm not sure on team creation limits and I don't think they are offering paid storage yet. Because of this, I'm also looking at IPFS[0] for cold storage and using other methods to catalog and index.
They added encrypted private git[1] registries and dropbox like file management[2]. That plus the mobile app, teams (groups), chat, and that it's primarily an identity provider I think it's a killer platform for file management and organization. It's still kind of immature in the chat and friends list area (not in a way that makes it unusable, just a little unpolished), but it's open source[3] and all of the functionality that is there works flawlessly for me so far.
Their recent edition of "bomb" messages[4], where you can send a private message with an expiration, I've found to be super useful. It has clients for all major operating systems and mobile platforms. Source is right there on github if you'd like to modify/enhance. You can even browse your git repos from your phone! They haven't added the ability to commit/push to git from mobile yet.
Now I don't have to worry that if I change a credit card and forget to update dropbox that I'll loose my stuff. Which happened this year. I didn't loose anything, but I realized I had not checked in on my over a decade worth of archives in a bit and hadn't paid them.
I took a look at Keybase a couple months ago and decided to make this move after hearing the Software Engineering Daily podcast with the found Max Krohn[5]. It's a pretty good walk through on their goals and current offerings.
Even with it's lack of commercial SaaS flair, I've actually found using it for chats, file sharing, git, and organization to be a fairly large productivity boost. Maybe because it's pretty no nonsense. I've moved half of my work dev chat to it from slack, and I don't miss Slack at all except for video chat/screen share. I especially don't miss it's the terrible file management / organization.
The next phase of my organization, I'm working on connecting it to a personal web portal where I can share and access control the content, but also curate my links, writing, and media I create in an HN/reddit style feed and use Kb to federate to other people's (friends) sites for content to and group that content by relationship and content category. So basically a smarter web ring (Everything old is new again!) with RBAC/circle of trust access control and collaborative versioned content editing.
I'm also exploring other similar tech in tandem or instead of Keybase like IPFS/Mastodon/WebTorrent. So far, for a lot of what I want though, I think Kb has it nicely tied up with a bow, though I doubt it'll scale to any interesting level of traffic without some elbow grease on the ops/storage side.
[00] https://keybase.io
[0] https://ipfs.io/
[1] https://keybase.io/blog/encrypted-git-for-everyone
[2] https://keybase.io/docs/kbfs
[3] https://github.com/keybase
[4] https://keybase.io/blog/keybase-exploding-messages
[5] https://softwareengineeringdaily.com/2017/10/24/keybase-with...
[0] https://github.com/keybase/kbfs#status
Also one thing which bothers me is online only mode, you can't cache the data offline and in Europe those speeds are far from great (around 20Mbps for download).
They have the weird community forum where employees don't post anything except for announcements. And now they have an extra tier before actual support, where your question is 'answered' by some community member or someone like that, who apparently can't be bothered to understand the question.
The Android app is a slow mess eating all available memory right away and/or refusing to edit notes. Inability to use SD cards is just what I want when the notes take 6 GB. And the website now “doesn't support” mobile browsers.
Pretty much the only thing where Evernote doesn't have better competition is the web clipper.
Other than the speed issues you mentioned, I'm also worried that evernote is eventually going to start selling my information (if it doesn't already) and I don't feel safe putting personal notes on it.
It doesn't really cover your file storage needs, but I think that's a different beast than notes and information. When I need to store stuff like that I dump it on Google Drive which makes search/sync/browsing dead simple/cheap.
The ability to expand/collapse, zoom in/out, and do everything via keyboard shortcuts makes it very pleasant to work with.
The outline paradigm is great, and Workflowy is a nice influx of fresh blood in that field, but I think I might actually kludge Org-mode into proper form with good mobile synchronization (and bonus privacy and customizability) before Workflowy makes any progress.
if you're using any Unix flavor, activate "locate" and use it. Both do an index of the filenames. So naming correctly your files is important. The 8.3 schema is over now ;)
On a WinPC I also use TotalCommander (of ghisler.com) to make an advanced search (filename and/or content). Of course, find&grep under Unixes do that well!
I got more than 40Gb of ebooks, pdfs, videos, podcasts, etc. in 4 languages (French, English, German, Spanish) and the sole way to find out quickly is global searching.
And constant reorganizing. (With no tool to update, it's a breeze ;))
If you prefer a hierarchical way, then using aliases (symbolic links, on unixes, but since Win7 you can also create links under windoze) can be an alternative to have multiple entries of the same files in your "database" of filenames, making possible to have "NoSQL for dummies" in "RDBMS" as well as "BigData" and "books for beginners" categories.
P.S. I work on documentation systems since 1984. Full text indexing is probably the best solution.
Madmarsu
I always enjoy reading this post about tag-based file organization, which feels like it would be a great foundation for this: https://www.nayuki.io/page/designing-better-file-organizatio...
For brain dumping text I put things into text files, named by YYYY-MM such as 2018-08.txt and they all end up in a notes/ folder.
Then I have a bunch of folders for things that are non-text, organized by whatever it is. I try to keep folder nesting at a minimum.
Searching text is super simple with grep and searching for specific files works well enough with file searching tools (and are also easy to browse manually).
For instance, my music resides at "Music/Artist/Year - Album/Track# - Tracktitle.Filetype", with appropriate additions to the Tracktitle to signify bonus tracks, remixes, live tracks and featured artists.
I've adapted pretty much the same system for all of my other documents. It's a bit old-school, but it works.
EDIT: I'm still trying to find a good machine readable way to represent the metadata for all that information. Alas, there are only so many hours in a day. Will probably end up with tags and Pinboard.
The main downside, as someone else has mentioned here with org-mode, is the lack of a good mobile editor. This is something I’ve wanted to find time to fix for years but still haven’t had a chance.
I sync to a server with vim and vimwiki installed and on my phone I just mosh[0] in. I don't generally do extensive editing on the go, though. Did the solution you have in mind include a specialized editor for touchscreens?
[0]: https://mosh.org/
Most information is stored in of 3 ways
1. Synced to a NextCloud instance running on a VPS.
2. Synced to a home NAS running Syncthing (though I may change this to IPFS at somepoint).
3. Added to Wallabag (A read it later like service)
These are put into Orgmode which syncs with Owncloud. I can then use the WebDav endpoint to pull the files to add or edit on my phone.
Articles/Papers
These get added to wallabag then are accessed via an RSS reader.
Personal Documentation
Added with Camscanner, OCRed, and added to NextCloud. At some point I will probably also store these on my NAS, so I can extracted the OCRed text as a text file so I can search them, but currently the collection is to small to justify this.
Media
My NAS handles most of this, and organizes it using Emby. Syncthing is my secondary means for this, and Emby watches the directory to organize this.
eBooks
Syncthing to add to the server. A headless calibre instance then organizes it.
Pictures
Most pictures come my from my phone, nextcloud will automatically sync these. If the pictures are part of notes, then instead they go into Orgmode, which works with images as well.
There are a couple of other organizational thing that use Syncthing + Organize, but nothing of major note.
This is not A single cohesive system but several scoped systems with a standardize way to add content to organize (e.g. folders). Searching the content is generally done via plain text searches, or inside of the application that manages that context (see emby & calibri).
Links
1. https://github.com/nextcloud/server
2. https://github.com/syncthing/syncthing
3. https://github.com/tfeldmann/organize
4. https://github.com/wallabag/wallabag
5. https://www.camscanner.com/
6. https://beorgapp.com/
7. https://manual.calibre-ebook.com/server.html
8. https://emby.media/http://onemodel.org (yes I plan to move to https sometime)
You can think of it something like a personal wiki + emacs org-mode, using postgres, but with a much larger vision than today's features, including sharing (linking/copying) between instances, and computability of the info. The most current code is in github (AGPL). Comments/questions very welcome, preferably via the mailing list; be patient if my answers are slow. Announcements list is low-volume.
Edit: it can store files, but isn't especially smooth about it (yet). For personal notes of all kinds, it is the most efficient, effective, flexible thing I know of.
Edit: the FAQs link to a discussion comparing it with emacs org-mode and others.
Edit: it has built-in text search, and some finicky but very functional import/export.
Based on this concept http://www.mandal-art.com/
but original mandalart author had/has less effective implementation
Truisms from time: - Content > tool config - Simple is better - Have good search - ISO 8601 date format, line based entries
Old and current flames over 20 yrs: - Google Desktop - TheBrain - MindManager - Freemind - NotationalVelocity - TiddlyWiki - VimWiki on vim as Deft folder on emacs in evil-mode - Org-Mode in native and evil mode - log bash script - sqlite3 - Traveller notebook, pencil and paper
In short, I find that writing them down every day in bite-sized chunks is most helpful. This method probably will not help you store non-texts such as movies, pictures, but for other mediums, it is great.
After about two years, the main benefits I find are:
* Bite-sized notes are easier to index and search for later
* We can easily share those notes with others when in a technical discussion etc, saving a lot of time
* We can do spaced-repetition on those notes easily because they are concise. (I wrote a short Go program to do this for me)
[[379]] Sjö Milljarður Manneskjur
| Tölvuleikur - Forritun
| Inventory - Steam library
| Computer Science - Concepts - Multiprocessing
|
| ,,Seven Billion Humans''
|
| 2018; re: concurrency
|________
| Backreferences:
| [[380]] Tomorrow Corporation
| > 2018: [[379]] Seven Billion HumansThere were some changes in version 3 and the subsequent rebranded Keep It[1] which kept me from upgrading, but it might fit your needs.
[0] https://web.archive.org/web/20130315081858/http://reinvented...
And Google Drive for the "larger” and better organized stuff like photos, e-books, or invoices.
I use meaningful titles and tags. That's enough.
I do organize my knowledge since years (http://metamn.io/beat, http://metamn.io/pulse, http://metamn.io/gust) and now I’ve collected these experiences into a Wordpress theme.
Still under development / test phase ... you are welcome to try it out.
MacInTouch More (1)
macOz Omnigraffle (2)
(1) https://en.wikipedia.org/wiki/MORE_(application)
(2) https://www.omnigroup.com/omnigraffle
Notes:
TreeSheets: Free Form Data Organizer
https://news.ycombinator.com/item?id=15057392
Quetree – A Q&A site that is tree-based and hierarchical
Downloaded data: Folders PDF, Video, TV. Own Generated data: Photos in a YEAR/MONTH Labels Textual: In Dropbox, Excel, Text, Word
All data on master Hard Disk, two more copies.
I also have passports scans and other sensitive stuff. I don't trust that I can secure mediawiki well enough to put those in clear on the internet (in upspin they are encrypted with a local key and the storage is my own server, mediawiki is in the public cloud).
One of the big features I like in Evernote is the ability to search pdfs and images. I have a lot of technical manuals that I save, along with receipts/etc that are easier to search for.
If your data is variegated in format and form, don't look for a single tool/method solution to organize them. There is no single all-purpose organization method.
Don't start with figuring out a method to organize your information. You will end up overengineering your org method.
Instead, organize your information per "use-case". Start with a specific use-case/project (e.g. writing a blog or paper), and work backwards to figure out how to organize your data to meet the requirements of that project. Do a couple of iterations. After a couple of projects, and you will naturally discover your own data use patterns.
If you go with a super organization system on day 1, it will likely be too general and require too much effort (tagging, keywords, hierarchies, version controlled, branches, etc.) you will end up expending resources on metadata management on data that you may never ever need to retrieve and very soon you will abandon the effort.
My PKM system is very simple: a single unorganized Google Docs for quick thoughts and ideas (just bullet points), separate Google Docs files for specific projects, etc. and Dropbox for files. It's simple, searchable, and multi-device.
I also occasionally use some specialized tools like Jabref (BibTeX) for specific types of data like references, but I hardly ever write papers anymore, so these have fallen by the wayside.
I've tried wikis but due to their multipage nature, they segment knowledge too finely (often there are wiki pages hidden in deep in the link hierarchy that I forgot existed). Wikis don't fit the PKM use case that well, so these too have fallen by the wayside for me.
For me, PKMs need to in some way feel like a single broadsheet where I can easily see and touch my information without having to drill-down hierarchies and follow too many links.
p.s. I've heard good things about Evernote. It's a little too heavy for me, but many people seem to find it useful.
I think this is an important point. I like to start with a few simple rules:
- To retrieve information, I should know where to start: a Schelling point.[0] For me, this is the home page of my wiki. For wenc, it's a Google Doc.
- It shouldn't take me more than three clicks to get from my starting point to the information I'm looking for.
- Links/URLs will tie everything together. They are the edges in my knowledge graph. But as wenc notes, keep the graph shallow.
Then I need to be rigorous, reorganizing things when they don't work intuitively and adding new nodes when something I need has not yet been recorded. As wenc puts it, "discover your own data use patterns."
Wikis do work for me, provided it's organized around Schelling points. I've used and refined these principles in setting up wikis at my last 3 companies and it's worked pretty well for organizing a collective knowledge base as well.
I see this sort of criteria and really don't agree. There are many operations that take more than three clicks. Navigation a UI is a graph in itself, and we can handle way more than 3 nodes. In the same way that I can remember how to get to work, or get a book from one of my bookshelves.
Anyway, small point.
I'm fairly settled on a homebrew style of noguci's system including pen, folded paper in my back pocket, a few paper notebooks as well as markdown, google docs, simplenote (via notational velocity) and google keep.
Here's the post that inspired me towards this system - http://www.literatureandlatte.com/forum/viewtopic.php?p=1592...
However, my system is stunningly poor at filtering and modelling information into memory. I tend to frequently write, bookmark, capture things and leave it at that. That does not help. I have recently begun exploring concepts of spaced repetition to this effect but its too early to comment on this.
I have learnt to differentiate between archive and active areas. Screens and the internet while great at archival, just dont work well for active information that I want to recall at will.
Just wanted to point out that important difference between active and archive information. That link I posted above was from my archival system but remembering Noguci's system and that it applies in this case is from repetition over active information systems.
1. Make sure your files and their organizational scheme can be transferred between tools. Your files need to be searchable, and at some point you will need portability. Text-based notes are easiest, but formats that have ubiquitous support (like docx or html) are ok too. Similarly, a collection of files is more reliable and flexible than a database tied to a subscription-based service.
2. Don't bother with elaborate tag or folder based taxonomies. Search is more efficient. I, at least, cannot accurately predict what taxonomy I'll need 5 years in the future, nor can I consistently tag every file correctly. There is some empirical support in favor of search over tagging [1].
3. Keep your notes organized; store your other files all over the place, wherever it is convenient. Use your notes to record context and thoughts relevant to a project/task/other activity, and use hyperlinks to connect those notes to the relevant files (wherever they are). Being able to embed or preview images in your notes is also very useful.
4. Organization is worthwhile for files that you have read/watched and thought about. The purpose of organization is to help you revisit and develop your thoughts on a topic. Organizing files you haven't read is not progress and does not improve your capabilities; you have to actually read and think about them to get any benefit.
5. Organization is also worthwhile for files that you know you will need for a specific project/task. Add a link to the relevant item in your notes for that project/task. Include a note why you thought it would be useful to future-you.
6. Files that you have not thought about / critiqued, and which do not fit a specific project/task, are not worth organizing. Dump them in an unorganized "to read later" list or just forget about them. The idea that you actually will get back to these files is mostly fictitious, and you can easily re-find them, or a superior equivalent, via search engines, browser history, or library logs.
7. Switch tools as needed based on circumstance. For example, my personal work notes are in org-mode, academic papers are in Zotero, personal work data is in a git + git-annex repo, and shared work notes and data are spread between Google Drive, Dropbox, and a private file server, depending on the requirements of the other people involved. I also have personal life notes in org-mode, Simplenote, and some ancient notes still to be migrated from Evernote. Hyperlinks tie files together as needed.
[1] https://people.ucsc.edu/~swhittak/papers/chi2011_refinding_e...
- things I want to (re)read next
- "best" / "mind-blowing" technologies, usually built around a simple, minimalistic, but powerful concept (e.g. reagents for composable lock-free data structures, parallel prefix sum, Futamura projections, etc)
For everything else full-text search in Zotero has served me well.
About portability between tools I would go with Pandoc, it supports conversion between a number of Wiki formats and markdown, which should suffice for most purposes. Definitely avoid tools that lock you in their proprietary formats.
Since you mentioned those specific examples, I see a commonality and so I'm going to assume you're primarily concerned about organizing files that are not authored/generated by you. (Except possibly pictures/photos since you might be talking about jpgs from your personal iPhone or digital camera.)
This distinction is key because software like Org-Mode, Evernote, MS OneNote, etc is better for personally generated note taking -- like classroom lecture notes, or brainstorming, or personal todo lists. Those tools are not suitable for saving and managing a terabyte of movie files. However, they have one exception to the "being bad at managing collections of the world's data that I didn't write" -- they do have very good "web clipping" functionality. Instead of saving url bookmarks that might suffer digital rot, you can use the web clip tool. (Personally, I prefer to save important webpages as .mhtml archives. This doesn't work for javascript heavy sites but for the type of informative webpages I want to save for later reference, this usually isn't an issue.)
Likewise, the software that's good at cataloging a "library" of collected files with capabilities for custom tagging (e.g. Adobe Lightroom for photos, iTunes for music, Zotero for pdfs, etc) are the wrong tools for archiving personal notes.
For a disparate collection of digital files, the best method I've found is to leverage the native filesystem in Linux/Mac/Windows. Create a hierarchy of well-named folders to build a sensible taxonomy and put all your pdf, epub, mp4, jpg, mp3 in meaningful locations. There is no good universal organizing software for organizing all the file formats that has the longevity and transparency of the native file system.
It's replicated from SSD to HDD on the same computer immediately (I use PostgreSQL replication and run multiple servers locally), so if one of those fails I lose nothing.
If I ever want gui or cli tool it's easy to hack up electron app or PHP/node cli script in an hour or so. DB access is much easier than parsing text files. It's easy to extend (add columns/tables) without messing with some random txt format.
You can also much more easily answer questions about your personal data by just firing up a psql and writing a quick SQL query with a much better chance of finding an answer to wider range of questions without resorting to actual programming/scripting.
You have to write some schema, so that naturally brings some organization to your knowledge as you have to think about it.
- https://news.ycombinator.com/item?id=17586375
One major pain point: no totally seamless mobile sync that I'm aware of. I just write things down in Google Keep and transcribe them when I'm on my laptop next.
Gordon Brander's patterns [2] (on HN recently) seems quite close to these and what you're after.
[0]: https://en.m.wikipedia.org/wiki/Memex
[1]: http://www.dougengelbart.org/about/vision-highlights.html
I ‘ve tried all kinds of ideas before settling for this setup, and they all felt forced or just too much trouble for what they were to me. Text files, Dropbox, and Spotlight have been a perfect combination for my needs for years now.
The thing that I find really makes things click is to figure out/set up good keyboard shortcuts and then practice them, so that you are able to save files to anywhere in your organiser hierarchy very very quickly. Say an average of five seconds per file. They say a 10x quantitative change is a qualitative change; once I had this down, I started using it a ton more and treating it kind of like an extension of my memory.
For Mac users some useful starting points for this technique are Default Folder X and ClipMenu.
TiddlyWiki is rather inscrutable at first, but extremely customizable and very thoughtfully architected. What I like most is that it is doesn't rely on an internet connection, it's very customizable and non-linear. The basic idea of being able to interlink things and create new pages just by adding brackets around a word is something I miss anywhere I don't have it.
I use TiddlyWiki in an Electron wrapper made with Nativefier. With a bit of custom CSS the interface is very pleasant.
By the way, you can do simple joins in Airtable by using the "lookup" and "rollup" field types, but I agree that it is relatively limited in that respect.
A lot of people think of GDrive as pure storage but unsurprisingly Google search within drive is pretty good at finding stuff. Although when you know it's there and search fails, it's frustrating as hell.
Or you could write the locations of your targets on the backs of turtles.
For notes, I found the majority of my note-taking is research and reasoning to support decision making for either software or business projects. For that I built my own tool to capture rationale make it useful:
you can organize the "media" think in "attachment" and tag them or create page about them, and you can still find the attachment in your file manager
"Zim is a graphical text editor used to maintain a collection of wiki pages. Each page can contain links to other pages, simple formatting and images. Pages are stored in a folder structure, like in an outliner, and can have attachments. Creating a new page is as easy as linking to a nonexistent page. All data is stored in plain text files with wiki formatting. Various plugins provide additional functionality, like a task list manager, an equation editor, a tray icon, and support for version control. "
https://zettelkasten.de/posts/three-layers-structure-zettelk... https://github.com/renerocksai/sublimeless_zk https://tiddlywiki.com/ http://journal.mcmorgan.org/view/incremental-paragraphs/ward... http://fedwiki.org/hack.platform.earth/fedwiki-for-notetakin... https://github.com/burtonator/polar-bookshelf
Perhaps a single system is not the answer, but a combination, e.g. for taking notes I'd pick fedwiki/sublimeless_zk/Zim/org-mode.
For organizing PDFs with full-text search Zotero is quite good, and you can link to your PDFs from your notes, but I wouldn't store my notes about PDFs inside Zotero (because then it'd be separated from my notes about non-PDF articles).
For keeping track of articles Firefox bookmarks (with Firefox Sync on Android acting as a 'read me later' bag).
It seems they have merged with readcube who want a yearly subscription however I was still able to buy a life time license to papers app a few months ago. I guess support will slowly disappear but for now its working great, with PDF sync via dropbox onto my iPad or iPhone, the ability to draw and take notes and search as well as download new papers. For your own PDFs I guess this would still work pretty well since you can search and annotate, create smart groups etc.
- It's like Workflowy, but better... - I've tried Evernote, OneNote, Apple Notes, Google Keep... they all just end up becoming a massive mess of unorganized notes.
- mobile and desktop friendly (especially input) - full local backup, including any metadata (so that I can always use another software, not cloud-first-with-export) - can be purely single-person (plenty existing sharing platforms) - it’s gotta support arbitrary nesting (I hate evernote/one note three levels to a wisiwig page rigidness) - any others?
I also wonder whether there’s enough market for selling this software (Airtable is going after business use cases, the single-player use case makes it quite limited).
Forgive the design, I’m right this very weekend updating the whole thing. I’m busy writing a little web app which will replace the spreadsheet you’ll currently need to keep track of your numbers.
Do you still find value in the enumeration aspect? Is this something where there's more value in it within collaborative environments?
Currently, I am trying StackOverflow Team but only for myself. I am so used to question and answer format that it makes it easy for me to dump the information. Most of my searches on Google ends up on StackExchange families of the website that this solution has that familiarity. Plus, it has some versioning built into it. It lacks on some fronts such as limits on the size of images. I am still looking for a better solution.
Can you expand on this? Was quiver not good enough at basic text?
I'm not affiliated in any ways but his ideas a really well thought out and apparently thoroughly tested with clients.
Personally I store raw thoughts and ideas in Keep, then transfer them daily or weekly to OneNote. Every so often I review the notes.
I have notes from anything important I've ever learned, ranging from technology stacks to economics to capital markets to psychology or personal well-being. I now never worry about losing thoughts or not seeing "the big picture" - it's a picture (note really) that I've saved over years now. And ever so gradually, I just build on that knowledge base, which I see as an extension of my own brain.
Edit: and if you don't use the cloud, _make sure you back up every few weeks_. It takes 5 minutes and you will really regret losing part of your brain if you break your laptop.
I find that any kind of knowledge search system eventually gets saturated and requires increasingly more time to scroll through partial hits to find what was asked. Our brains work the fastest in associative mode, it'd be nice to have some kind of associative storage (other than tags) that is personally biased/customized.
Reviewing the kept info is one way to refersh these associations in our organic minds.
But it'd be cool to ask your 'MindAssistant' to retrieve you the whole context of your recent or old idea, complete with your thought pathways: 'Hey, Mind, please remind me what was I thinking about the shapes of the clouds in the sky', '-Here's your Mindmap, thought-journey, and most important insights, Sir. At the conclusion of these thoughts, you read the following articles via HN...'
For now, generically speaking, I use OneNote for keeping track of my own generated ideas and pasting in things from other sources. For larger media I recommend having a network drive or using Dropbox.
I'm happy with OneNote.
For my more technical ideas where I'm brainstorming I have a nice leather journal (no lines) that I use for brainstorming. I tried using a Surface Studio as well with the pen and that works good too because you have an infinite canvas and can move things around. With paper / whiteboard brainstorming I frequently run out of room. Digital means I can just drag a bunch of stuff to make more room. Plus having the pen and choosing different colors makes it really freeform.
Let me warn you: I tried to use OneNote this way. The web clipper is really nice and I used it heavily to clip everything interesting on the web from my desktop and my phone until my OneNote file reached 8GB. Syncing just completely gave up and there is no way to export everything in a usable format.
Since then I use a separate Firefox 52.5.0 instance with the old ScrapBook plugin. I can send sites to it from a current Firefox instance and from mobile. It works ok and, in case this setup should stop working at some point in the future, I still have all sites sitting on my local drive.
Unfortunately the browser makers don't give a damn about saving and organizing Websites locally, although I'd believe that this would be far superior to Bookmarks. I love scraping my local library for useful stuff and I've been bitten by link rot far too often.
For personal notes and todo lists I prefer simple text files with a few categories. Lately I'm also giving Boostnote [1] a try, which simply looks a bit better and allows me to add checkboxes and proper headings/separators.
[1] https://addons.mozilla.org/en-US/firefox/addon/archiveror/
Since it supports hyperlinks to a wide variety of data, and Emacs is so easy to script, you can definitely do that. For example, org-ref has a lot of functionality implemented to organize academic papers, including the corresponding PDFs.
In fact, a personal org-mode wiki could easily have one or more files devoted to personal pictures, books, papers and hyperlinks to those plus whatever metadata. There are some examples around if you google for that.
I do keep my personal wiki very close to a zettelkasten, but implemented in org-mode. Nothing special. Very simple. Small files that mimic cards, plus hyperlinks. I search using ripgrep, and I version control using git. A key feature is that I keep my task list and calendar inside my wiki too, but not version controlled.
Are there services for this where you can link in your own backend?
Have you written anything about it before, I'm curious about more specific examples around what you're doing.
Do you automate most/some of what you put in?
I've thought about writing some scripts to export things from various online accounts/services/apis and importing them into a personal database mostly for backup purposes. It sounds like you're already doing that, and more.
- Some of the data import is for archival purposes and mostly one-shot. I'm old enough to have seen many services where I put my data in, or where I had interesting conversations, go and delete everyhing in the process.
- Some of it is for ease of access and offline use. For example Jira can be a very slow system, where each action takes 10 seconds to complete. Some services have too bloated an UI to be useful on slow computers. You can easily lose access to some data/technical comments if permissions change, etc. If you have data locally, it's possible to avoid all those problems and present the data in any way you want.
Yes, I automate imports from external services. In the past I used PHP, now I mostly use Node or Electron, because it has a transparent support for binary data and JSON columns in postgresql, and if I want to parallelize HTTP requests it's easier to do in Node.
Import/sync scripts always have the same structure. First there's some way to gather the data from the external service, then I import it into DB with update/upsert/insert helpers, for example:
upsert(db, 'table_name', {id: 123, name: 'qwe', ...}, ['id']);
One other method I use, with web services that are too messy or complicated, is to create a userscript for Violentmonkey that gathers data as I browse the website and posts it to the database via a small localhost http service. This is useful for JS heavy websites that don't have nice JSON APIs. The script just runs on the background (or can be triggered by a keyboard shortcut) and uses the current state of the DOM to get the data and send them to the database. This method of import is invisible to the service itself.
Most of the scripts are fairly simple (< 100 lines), because most services are built around ~2 interesting primary entities that are worth storing.
Because I use upsert most of the time, I can and I sometimes do add some extra columns for annotation purposes. For example I mentined Jira. I added columns for ordering issues and for marking them with whether I wait for some feedback or not, and then I have a simple Electron app for ordering and listing my issues that I need to work on where issues that wait for external feedback are not shown until the feedback is provided. It's uncluttered, and as fast as you can imagine, being backed by a localhost database.
In Node I use: https://github.com/request/request-promise-native and https://github.com/request/request and https://github.com/vitaly-t/pg-promise
PostgreSQL replication is well docummented elsewhere. I replicate the entire cluster, which doesn't require any maintenance when creating databases/tables, so it's absolutely painless after the initial setup. All you need to do is check the logs from time to time. All I do is that I run the backup database server on a different UNIX socket and disable the TCP/IP interface. It is very flexible, as you can also put the backup server on a different machine, you can have multiple backup servers, chain them and it just works even if you have intermittent connectivity between the machines.
Most of my system is not universal or systematic and is very specific to my use cases. But that will be true for anyone wishing to preserve some of their data in a world that is fairly hostile to data portability. In fact, lack of universality/configurability reduces the complexity by a lot and makes it all easily manageable for a single person with at most a few hours/data source of time investment.
For iOS another possible option is Editorial. There are some script addons convert orgmode to MD ↔ ORG. This is incredibly hacky though, and I'm unsure how it works.
I use beorg but it still needs a lot of work to be what I'm looking for honestly.
Occasionally I have to consolidate multiple edits with ediff-files, but that's normally only after I have accidentally disabled wifi on the phone for longer periods of time.
On top of this, if you read a fair bit of papers, check out org-ref [1]. Referencing and notetaking of papers becomes a breeze.
Finally, you can also export your notes to a bunch of different formats (e.g. html, pdf).
- Personal (health, finance, hobbies, spirituality, etc)
- Work (things for my company, invoices, PDFs for learning tech stuff, etc)
- Church (I volunteer for a couple of churches, so that's very misc)
The exact same structure exist for my Google Calendars (separated by spheres that encompass several projects/concerns).
I wish more people would do that as link rot is a very real thing, certainly for technical information often saved in gists or github repos that are often harder to find back.
https://www.mediawiki.org/wiki/MediaWiki
Personally, I've been using WikkaWiki for years:
https://github.com/bakoontz/WikkaWiki
I would prefer something that supports markdown. Neither of these really do.
https://www.lesswrong.com/posts/reitXJgJXFzKpdKyd/beware-tri...
There was a joke in one of Stephen Hawking's book that each equation included in the text cuts the number of readers by half. In a similar fashion, you can imagine that each extra step you need to get to the information will cut the number of occasions you do so by half. If you want the system to benefit you daily, in many areas, it has to be as simple and seamless as possible.
Underlying idea: friction disincentivizes.
I would not expect you could get a Nobel Prize for that, though. Maybe Scott Alexander should start submitting his articles wherever it is you need to submit them to get a Nobel in economics.
Proving that all the other economists are modeling things wrong is more impressive than you make it sound.