Ask HN: Has Anyone Trained a personal LLM using their personal notes?

138 points by Erazal 2 years ago | 69 comments

Some here comment on notes they've taken for their entire life, often using powerful note-taking systems.

Of course, there's a ton of note-taking systems out there. Org-Mode [1], Obsidian [2], plain .txt, ...

And it's become quite simple to integrate such systems with LLMs.

Whether to add that data to the LLM [3], using LLM formatting, or to visualize and use it as a personal partner. For the latter - there's also a ton of open-source UIs such as Chatbot-ui[4] and Reor[5].

And that's just the tip of the iceberg.

Personally, I haven't been consistent enough through the years in note-taking.

So, I'm really curious to learn more about those of you who were and implemented such pipelines.

I'm sure there's a ton of cool interaction experiences.

[1] https://orgmode.org/ [2] https://obsidian.md/ [3] https://ollama.com/ [4] https://github.com/mckaywrigley/chatbot-ui [5] https://github.com/reorproject/reor

KMnO4 2 years ago |

Maybe not exactly what you’re asking, but I started doing talk therapy last year. It’s done virtually and I record the session with OBS. As soon as the recording finishes, the following happens:

- The audio is preprocessed (chunked) and sent to Whisper to generate a transcript

- The transcript is sent to GPT-4 to generate a summary, action items, concepts introduced with additional information

- The next meeting’s date/time is added to my calendar

- A chatbot is created that allows me to chat with each session, including playing the role as the therapist and continuing the conversation (with the entire context of what I actually talked about)

It’s been exceedingly helpful to be able to review all my therapy sessions this way.

rhcom2 2 years ago | |

I'm sincerely happy you're finding value in this and it's a very impressive workflow. The idea of sending my therapy sessions to OpenAI sounds terrifying though.

kid64 2 years ago | |

I hope you're the patient in this scenario, otherwise this is an egregious HIPAA violation.

mmahemoff 2 years ago | | |

Might still be a violation if they're the patient? Unless therapist and their employer's consent is given and ofc dependent on the relevant jurisdictions (IANAL).

fullautomation 2 years ago | |

This is really interesting, are your comfortable with OpenAI having your personal details in this case?

Erazal 2 years ago | | |

This is where having our own LLMs and stacks running locally will save and empower us IMHO

tqwhite 2 years ago | | |

OpenAI's privacy claims are fine. I wouldn't worry about this any more than I worry about my email provider.

Erazal 2 years ago | |

wow - really cool.

I'm actually the founder of an AI Meeting Bot company - and we're thinking of open-sourcing so you could run exactly this set-up locally with perfect diarization / recording while also maintaining privacy [1].

I'm currently creating code examples, and just finished the "chat with each session". Would love to know how you implemented it.

[1] https://aimeetingbot.com

iamcreasy 2 years ago | |

Did you discover anything interesting by being able to review all the therapy sessions?

NicoJuicy 2 years ago | |

Curious on the code. ( a friend is a psychiatrist and she noticed difficulty with multiple languages and device translations).

This flow could help me improve fluency in her sessions ( eg. she has a hardware translation device (expensive) which has significant issues auto translating ), since it's missing context a lot.

Eg. When grieving is incorrectly translated between dutch-polish, it defeats a bit of the purpose of being fluent in your native language.

Reducing the error rate would help a lot.

psychopomp 2 years ago | |

I’d love to replicate your workflow. Any luck with speaker diarization using whisper? I’ve tried WhisperX several but it didn’t work.

Erazal 2 years ago | | |

I've created an AI Meeting Bot API to do just that [1].

At the moment it runs on AWS, and we're thinking of open-sourcing so you could also run it locally to maintain 100% privacy of such conversations.

You'd get speaker diarization, names on top of the recording [2].

[1] https://aimeetingbot.com [2] https://spoke-1.gitbook.io/ai-meeting-bot

Happy to get in touch and have you run it

appstorelottery 2 years ago |

I've been disciplined (perhaps obsessive at times) with keeping a daily diary for many years and I was interested in being able to query my diary locally via AI. I found a solution that works surprisingly well using GPT4ALL.

I found GPT4ALL (https://gpt4all.io) to have a nice-enough GUI, and it runs reasonably quickly on my M1 MacBook Air with 8Gb of ram, and it can be setup to be a completely local solution - not sending your data to the Goliaths.

GPT4ALL has an option to access local documents, via the Sbert text embedding model (RAG).

My specific results have been as follows; using the Nous Hermes 2 Mistral DPO and Sbert - I indexed 153 days of my daily writing (most days I write between 2 and 3 thousand words).

Asking a simple question like "what are the challenges faced by the author?" provides remarkable, almost spooky results (which I won't share here) - which in my opinion are spot-on regarding my own challenges over that the period - and Sbert provides references to which documents it used to generate the answer. Options are available to reference an arbitrary number of documents, however the default is 10. Ideally I'd like to have it reference all 153 documents in the query - I'm not sure if it's a ram or a token issue, however increasing the value of documents referenced has resulted in machine lock-ups.

Anyhow - that's my experience - hope it's helpful to someone.

brotchie 2 years ago |

Played around with fine tuning, but ended up just experimenting with RAG.

One thing I haven’t worked out yet is the agent reliably understanding if it should do a “point retrieval query” or an “aggregation query.”

Point query: embed and do vector lookup with some max N and distance threshold. For example: “Who prepared my 2023 taxes?”

Aggregation query: select a larger collection of documents (1k+) that possibly don’t fit in the context window and reason over the collection. “Summarize all of the correspondence I’ve had with tax preparation agencies over the past 10 years”

The latter may be solved with just a larger max N and larger context window.

Almost like it’s a search lookup vs. a map reduce.

jdthedisciple 2 years ago | |

Interesting, been waiting for some free time to do this myself.

Mind sharing how you set up your RAG pipeline and which (presumabely FOSS) components you incorporated?

dleslie 2 years ago |

I want this for my photos.

I'm not a good photographer, but I have taken tens of thousands of photos of my family. I would love to provide a prompt for a specific day and persons and have it create a photo that I never was able to take. I don't mind that it's not "real" because I find photography to be philosophically unreal as it is. I want it to look good, and inspire my mind to recreate the day however it can imagine.

And I want to do it locally, without giving away my family's data and identity.

TuringNYC 2 years ago | |

This is basically what all the headshot generator apps do. It is pretty simple to achieve if you can spin up GPU instances (https://huggingface.co/docs/diffusers/v0.13.0/en/training/te....)

However, I find it challenging to achieve on Macbooks, despite all the neural core horsepower I have. If anyone has achieved this with non-NVIDIA setups i'd love to hear!

em-bee 2 years ago | |

interesting idea.

completely unrelated i just read a scifi story where a technology was developed that could revive dead bodies for a short while in order to pose for family photos that they hadn't created before the person passed away.

https://clarkesworldmagazine.com/liu_03_23/

obviously going way overboard for something AI can do today, but probably the author wrote the story before that.

cjbprime 2 years ago |

Going to wait for longer context local models. Fine tuning/training is lossy compression of your notes into the model weights -- there isn't much value in a vaguely-remembered copy of some of my notes. This is why other comments are pointing you towards Retrieval Augmented Generation instead, where the relevant notes are losslessly added to the prompt.

Erazal 2 years ago | |

I think you're right

jki275 2 years ago |

PrivateGPT is a nice tool for this. It's not exactly what you're asking for, but it gets part of the way there.

https://github.com/zylon-ai/private-gpt

2rsf 2 years ago | |

I used Private GPT with our internal markdown based developers portal, the results are OK-ish but are closer to a fancy search then to a chat.

dumbmrblah 2 years ago |

I’ve tried several different systems, nothing really stands out.

That being said, I’m trying to document as much as my life in anticipation of such programs existing in the near future. I’m not going overboard, but for example, I wouldn’t really keep a personal diary, but now I try to jot down something every day, write down my thought processes on things, what actions were done and why.

I’m looking forward to a day where I have an AI assistant (locally hosted and under my control of course) who can help me with decision-making based on my previous actions. Would be neat to compare/contrast how I do things now, compared to the future me.

yyytolys 2 years ago | |

This idea is also what I am thinking about. At present, I am also trying to do applied research in this aspect. If it can achieve substantial help, it will be very meaningful.

pb060 2 years ago |

Gianluca Nicoletti, an Italian journalist, writer and radio speaker, is training a LLM with all writings as a support for his autistic child for when he won’t be here anymore. The software will speak with his voice.

https://www.lospessore.com/13/07/2023/una-chatbot-per-contin...

woodpanel 2 years ago | |

Very interesting.

Since presumably "all writings" refers to all his writings during his lifetime, I'd hope it can account for those times in his life at which he changend his mind on certain topics?

pb060 2 years ago | | |

Yes I guess that “all writings” (sorry I meant all of his writings - not an English native) means everything he wrote and also his speeches, taking into account the times when he changed his mind. Knowing how meticulous he is I would be surprised if this wasn’t the case. Worth considering that he was recently diagnosed a mild form of autism himself.

Brajeshwar 2 years ago |

Neither mine nor endorsing, and I haven't played besides the initial installation but Khoj has an open source offering for this. Check it out https://khoj.dev

sabaimran 2 years ago | |

Hi! Founder of Khoj, happy to add more context.

Khoj will allow you to plug in your Obsidian vault or any plaintext files on your machine or Notion workspace. After you share the relevant data, it creates embeddings and uses it for RAG, so you get appropriately contextual responses with your LLM.

This is the best place to start for self-hosting: https://docs.khoj.dev/get-started/setup

jondwillis 2 years ago |

I started fine-tuning GPT-3.5 on a decently large corpus of my text messages and emails, and it pretty much generated schizophrenic output. I don’t think I did a very good job of curating the text that ended up fine-tuning, and I want to try again.

CSMastermind 2 years ago |

It sounds like you want RAG instead of training or even fine tuning a model.

Have you looked into the OpenAI APIs? They make it relatively easy to do assuming you have some limited programming knowledge.

Erazal 2 years ago | |

I'm currently looking to implement RAG locally, using QDrant [1] for instance.

Just playing around for now, but it makes sense to have a runnable example for our users too :) [2].

[1]. https://qdrant.tech/ [2] https://aimeetingbot.com

jdthedisciple 2 years ago | |

I have recently briefly looked into Assistants API (if that's what you're referring to) but it seems relatively if I'm not mistaken.

jondwillis 2 years ago | |

Are you referring to the "Assistants" API?

CSMastermind 2 years ago | | |

I was not, that's relatively new, though if I'm not mistaken it might make the process easier.

I mean calling the embeddings API and then having software locally that finds and appends documents to your queries.

alexmolas 2 years ago |

https://eugeneyan.com/writing/obsidian-copilot/

Erazal 2 years ago | |

hadn't seen your repo yet [1] - adding it to my list right now.

Your blog post is really neat on top - thanks for sharing

https://github.com/eugeneyan/obsidian-copilot

bhupesh 2 years ago |

Folks at gitbook are kind enough to give me a LLM over my notes https://til.bhupesh.me

Erazal 2 years ago | |

oh damn, I'm using gitbook too actually - for our users [1]. Not open-source but will definitely try it out ASAP.

[1] https://spoke-1.gitbook.io/ai-meeting-bot

phantomathkg 2 years ago | |

Do you have a Pro plan?

ein0p 2 years ago |

Somewhat related, for those of us who don’t take extensive notes: are there nicely packaged plugins for RAG in email, especially for eg Outlook or Apple Mail?

bredren 2 years ago |

Has anyone seen or used something that can train on a complete imessage history?

Presumably, I have more than enough messages from me along with responses from others to chat with a version of myself that bears an incredible likeness to how I speak and think. In some cases, I'd expect to be able to chat with an LLM of a given contact to see how they'd respond to various questions as well.

bshaughn 2 years ago | |

I've implemented a POC on exactly this and am working on something more sophisticated right now. Can I reach out to discuss more?

bredren 2 years ago | | |

Yes, I'd gladly trial something like this.

It must run locally / require no network requests. I can run on an M2 w 24GB or M3 with 36GB.

My email is in my profile here.

joenot443 2 years ago | |

Came here to ask the same question. I've played around with a few implementations, but nothing with results that were close to useful.

Anyone had success RAG-ing a chat history??

prashantsengar 2 years ago |

Not on my notes, but I have used GPT4All to chat with the documents of dapr. I downloaded the md files from the docs GitHub repo and loaded the directory in GPT4All.

It is not "training" a model but works pretty great.

Zambyte 2 years ago |

I have a large org-roam note system. I would like to create a pipeline where I can ask natural language questions, and it will build SQLite quries to efficiently crawl through the database and find what I want. I haven't gotten around to it though.

bshaughn 2 years ago | |

Not exactly what you're looking for but I a few months ago I spent a day building a llama-index pipeline against my markdown notes with a really privative note crawling implementation, and had surprisingly good results for question answering.

I don't use an org-roam note system but I've been working on a similar and highly opinionated note system that I'm always making tools for. And I'm always interested in seeing people's ideal note systems.

my crude WIP Obsidian / Markdown note RAG tool: https://github.com/bs7280/markdown-embeddings-search

mitjam 2 years ago |

I have heard good things about the Notion AI addon although I haven’t tried it myself.

Erazal 2 years ago | |

have never found it powerful enough - or I'm just setting too high a bar?

I have a ton of databases on Notion (with all my teams conversation transcripts, meeting to-dos, etc.) and global AI search just isn't there.

I haven't found a way there (but have elsewhere using open source) to create a kick-ass search.

bsima 2 years ago |

I’m literally working on it right now, dm on X if you wanna pair or something

Erazal 2 years ago | |

I just DM'ed you.