Show HN: 35k+ paper psychedelic library that knows LSD from Lumpy Skin Disease(consciousnesslibrary.org) |
Show HN: 35k+ paper psychedelic library that knows LSD from Lumpy Skin Disease(consciousnesslibrary.org) |
I just finished my master's in Psychedelics and Consciousness Studies, and built the library I wanted while studying. I started it over a year ago, but put it down until my practicum advisor suggested I dust it off and go hard on it during my last term.
As of this writing, there are over 35,700 papers by over 82,200 authors organized into 37 topics, growing daily. There's a 2D map of the whole corpus and evidence syntheses on the topic pages, plus the ability to generate your own syntheses with any prompt.
It's built with Rails 8.1, uses one Postgres (for full-text search, vectors, and job queue), and runs on a $24/mo DigitalOcean droplet that I just upgraded from the $12 tier and deploy to with Kamal. Total LLM spend so far is ~$28 in DeepSeek-V4-Flash, trending down after a big one-time backfill push.
Relevance was the hardest part to get right by far. Ensuring we keep papers on LSD the psychedelic vs Lumpy Skin Disease, or Ketamine for depression vs anesthesia in cats was no small feat. The design ended up being a cheap keyword prefilter, then an LLM rubric that accepts or rejects papers. More on that here: https://consciousnesslibrary.org/docs/article-pipeline.html
It's free, no ads, no signup to read anything, and a registered 501(c)(3) I fund myself for now. I'd really like to know what breaks or any other feedback, and would love to answer any questions about it!
Curious why you chose Rails for this? I'd think Python tooling would have been a better fit given the data-heavy pipeline doc you linked.
Also, I saw the https://consciousnesslibrary.org/ai-ethics link in the footer. Lots of ambitious goals there. Do you really think it's possible to run models in-house that can do what you're having deepseek do right now, especially with "renewable energy wherever you can source it?"
As for why I chose Rails - I just know Rails really well. I've been using it since 2012 or so, and it's basically muscle memory now. The most intensive parts of the app really would lend themselves to Python tooling and I did end up hand-building a lot of the data engineering stuff that there for sure exist plugins for in Python, but the trade off of using what I know well outweighed that by a lot. Plus the rest of it are perfectly CRUD-able server-rendered pages. Pretty Railsy aside from the data eng stuff.
I also started this before AI coding harnesses were available. I shelved it for a long time after the initial few rounds, but the work I did in the beginning was substantial enough to not just start over. If I had Claude Code with Fable when beginning, it might look a lot different. But I do like knowing everything about what I built and understanding every line.
On your AI ethics question... I do think it's possible, eventually. That's why I hedged a bunch on it being a long term goal. If open weight models gain steam and hardware acquisition becomes manageable, it would be a dream to run a self-hosted self-trained model in my house (which is powered by solar, so the renewable thing is real). It is especially realistic if some funding comes through for hardware.
Also, any plans to include trip/experience reports? I'm particularly curious about experiences regarding meditation and dreams and that side of what you have in the project. I know there are places like Erowid and whatnot for psychedelic reports but haven't heard of something similar for the rest of what you have in here.
On experience reports - I decided to keep it purely academic at first but will soon be working on integrating data from a tool by Josie Kins (the maker of https://effextindex.com).
I also don’t know of anywhere that aggregates reports from meditation and dreams etc, but that’s a really interesting idea. Maybe this can be the place!